The Gateway That Never Forgets #1881
Summary
A devshard gateway writes one SQLite session file per escrow, $DEVSHARD_STORAGE_DIR/escrow-<id>/epoch_<N>.db (template path /root/.devshardctl),
These files are never deleted. Each epoch adds several gigabytes.
DEVSHARD_STATS_RETENTION_EPOCHS only prunes rows in accounting.db.
Motivation
A full disk stops the gateway from opening new session files. On a production gateway the session files from epoch 403 through the epoch 409 occupied 37.8 GB.
| Epoch | Directories | epoch_N.db |
|---|---|---|
| 403 | 393 | 4.4 GB |
| 404 | 427 | 7.3 GB |
| 405 | 340 | 6.3 GB |
| 406 | 298 | 6.4 GB |
| 407 | 387 | 6.6 GB |
| 408 | 273 | 5.2 GB |
| 409 | 131 | 1.6 GB, epoch still running |
| Total | 37.8 GB |
A completed epoch typically leaves behind around 5–7 GB. Given enough epochs, the ending is predictable: the disk fills up.
Impact
- Who is affected: gateway operators.
- Effect: each gateway's local disk.
- Likelihood: common over time.
- Severity: high.
- Affected components:
devshardctlgateway storage (devshard/cmd/devshardctl/gateway.go,devshard/storage).
Expected behavior
On epoch change, the gateway removes session directories older than the host horizon (current + two previous), with the same skip for anything still live.
🔄 Auto-synced from Issue #1881 every hour.