The Gateway That Never Forgets #1881
Summary
A devshard gateway writes one SQLite session file per escrow, $DEVSHARD_STORAGE_DIR/escrow-<id>/epoch_<N>.db (template path /root/.devshardctl),
These files are never deleted. Each epoch adds several gigabytes.
DEVSHARD_STATS_RETENTION_EPOCHS only prunes rows in accounting.db.
Motivation
A full disk stops the gateway from opening new session files. On a production gateway the session files from epoch 403 through the epoch 409 occupied 37.8 GB.
| Epoch | Directories | epoch_N.db |
|---|---|---|
| 403 | 393 | 4.4 GB |
| 404 | 427 | 7.3 GB |
| 405 | 340 | 6.3 GB |
| 406 | 298 | 6.4 GB |
| 407 | 387 | 6.6 GB |
| 408 | 273 | 5.2 GB |
| 409 | 131 | 1.6 GB, epoch still running |
| Total | 37.8 GB |
A completed epoch typically leaves behind around 5–7 GB. Given enough epochs, the ending is predictable: the disk fills up.
Impact
- Who is affected: gateway operators.
- Effect: each gateway's local disk.
- Likelihood: common over time.
- Severity: high.
- Affected components:
devshardctlgateway storage (devshard/cmd/devshardctl/gateway.go,devshard/storage).
Expected behavior
On epoch change, the gateway removes session directories older than the host horizon (current + two previous), with the same skip for anything still live.
💬 Comments (2)
Hey @danielhubersstorm, it seems like @qdanik has already fixed this issue in #1905. But if you are open to contributing to Gonka, please read https://gonka.ai/docs/bounty-program/, and I hope you will find something interesting.
🔄 Auto-synced from Issue #1881 every hour.
Hi, I’d like to work on this issue. Is it still available, and does the work in #1905 overlap with the session-file cleanup?
My proposed approach is to clean up expired session storage on epoch changes, retaining the current epoch and two previous epochs while protecting live sessions and any state still needed for settlement or recovery. I would include regression tests for retention boundaries, active-session protection, and repeated cleanup.
Could you confirm the preferred target branch and any additional acceptance criteria?
Also, is this task eligible for a contributor reward? If so, I’d appreciate clarification on the proposed amount, payment currency, and approval process before starting substantial implementation.
Thanks!