[BUG] devshardctl: one escrow's runtime build error at startup stops the whole gateway #2034
Summary
At startup buildGatewayRuntimes builds a runtime for every active escrow and classifies a build error by name. Four errors deactivate the escrow and let the gateway start; ErrParticipantNotFound skips it. Every other error is fatal for the process: log.Fatalf("create runtimes").
The effect is not limited to the escrow with the error. A transient chain call, a damaged SQLite file, or a recovery error of one escrow stops the process, and with it all of its escrows at once, including the healthy ones that have nothing to do with the error: their clients get no answers, their Finishes are not delivered, their settlements do not run. With restart: unless-stopped the container loops, and the healthy escrows stay unavailable for as long as the one broken escrow keeps the gateway from starting.
The settlement of an escrow that the gateway cannot serve is a separate issue: #2039.
Motivation
Pre-existing on main (main.go:560-593 at fd99f88d). v6 (#1792) raises the exposure: shared Postgres, many escrows per gateway, 16 GetHostInfo calls per escrow over peer RPC, and new recovery steps (validation-obs and sealed-index rebuild). The allow-list grows by incident: #1988 item 9 added ErrParticipantNotFound after one missing participant record disabled a funded escrow.
Impact
- Who is affected: gateway operators and their clients; hosts in the affected escrows.
- Scope: one gateway, but total for it. The gateway is the only sequencer of its escrows: no requests, no Finish delivery, no finalize, no settlement while it is down.
- Likelihood: intermittent. One timeout in N × 16 chain calls at boot is enough. With
restart: unless-stoppedthe container loops and repeats the calls on each restart. - Money: an escrow that is not settled in its epoch or the next cannot be settled at all; two epochs later the chain splits the full deposit equally per slot (
devshard_pruning.go,distributeUnsettledEscrow). The owner gets no remainder back, hosts are paid as if the escrow was fully used, and no host stats reach the epoch record. - Affected components:
devshard/cmd/devshardctl/main.go(buildGatewayRuntimes),devshard/user/httpsession.go,devshard/user/recover.go.
Detailed description
Code at v6 head 8f9279f1 (same structure on main):
- main.go:591-634: classification; main.go:478: log.Fatalf. No retry, no backoff.
- Per-escrow errors that reach the fatal branch: a transient chain error in GetEscrow (httpsession.go:64-66) or in any GetHostInfo of the 16 slots (:118-122); open storage for that escrow's SQLite (:88-90); every error of RecoverSession (recover.go: group mismatch :153, version mismatch :161, snapshot restore :212, backfill diffs :294, sealed-index rebuild :390, validation-obs rebuild :420).
- These errors have different scopes: some are process-wide (chain or store unavailable, no key), some are permanent for one escrow (damaged journal, group or version mismatch), some are transient for one escrow (one slow chain call, SQLite busy). The classification does not distinguish them.
Reproduce:
1. Run a gateway with two active escrows.
2. During startup make one GetHostInfo fail (block the chain gRPC for a few seconds) or damage the SQLite file of one escrow.
3. The gateway exits with create runtimes: ... and never serves the healthy escrow; the container loops.
Expected: the healthy escrow serves; the failed one is reported.
💬 Comments (1)
🔄 Auto-synced from Issue #2034 every hour.
I solved these problems from this issue in #2035:
restart: unless-stopped.devshard_gateway_startup_skipped_escrowwith reasonruntime_build_failed, plus a log line) and is retried in the background every 30 s until it starts or a final reason deactivates it. No restart is needed.