Node resync from snapshot caused missed inference tasks due to large application.db #527
Hi,
I encountered an issue with my node where the application.db grew too large. Because of this, I had to stop the node and resync it from a snapshot.
However, during the resync period, the node missed a significant number of inference tasks. I would like to ask:
Is there any way to recover or compensate for the missed inference tasks?
Or is there a recommended approach to avoid losing inference tasks when a resync is required due to a large application.db?
Any guidance or best practices would be greatly appreciated. Thanks in advance for your help.
💬 Comments (1)
🔄 Auto-synced from Issue #527 every hour.
Hi @bingcongxihaha! Unfortunately, no inference tasks that are missed while a node is offline (e.g. during resync) cannot be recovered or compensated retroactively. Inference assignment and PoC are performed in real time. If a node is not running and serving requests during that period, those inference opportunities are simply lost.
The goal is to prevent forced resyncs by controlling database growth and disk usage.
Cosmovisor creates a full backup of the .
inference/datadirectory during upgrades. Make sure sufficient disk space is available. If disk usage is high, older backups in.inferencecan be safely removed. Largeapplication.dbfiles can be reduced using these techniques.