[BUG] devshardctl: an escrow the gateway cannot serve is never settled, and the chain splits its deposit after two epochs #2039
Summary
An escrow that the gateway cannot serve is excluded from routing but is not settled. The gateway deactivates an escrow when its runtime build ends with a final reason (ErrLocalStateUnrecoverable, ErrEscrowNotFound, ErrEscrowSettled, missing private key); after #2035 it keeps retrying an escrow whose build fails for any other reason. In both cases nothing settles the escrow. If nobody settles it by hand before the next epoch ends, the chain distributes its whole deposit equally per slot, and the owner gets nothing back.
Split out of #2034, which now covers only the gateway start.
Motivation
Pre-existing on main. v6 (#1792) raises the exposure: many escrows per gateway, and new recovery steps (validation-obs and sealed-index rebuild) that can leave an escrow unserviceable.
Impact
- Who is affected: escrow owners (the deposit), hosts of the escrow (no host stats reach the epoch record), gateway operators.
- Money: an escrow that is not settled in its epoch or the next cannot be settled at all; two epochs later the chain splits the full deposit equally per slot (
devshard_pruning.go,distributeUnsettledEscrow). The owner gets no remainder back, and hosts are paid as if the escrow was fully used. - Affected components:
devshard/cmd/devshardctl(deactivation and settlement paths),devshard/user/recover.go, and the host side if the settlement data has to come from the hosts.
Detailed description
- A deactivated escrow (
SetDevshardActive(false)) keeps its data and can be brought back by the admin endpoint, but nothing settles it automatically. - For the four final reasons the gateway has nothing to settle with: the escrow is gone or already settled on chain, the signing key is missing, or the local journal is proven inconsistent. In the last case the data needed for settlement (last state root, host signatures, host stats) exists only on the hosts.
- An escrow whose build fails permanently for a reason outside the four (for example a group or version mismatch in
recover.go) is retried and reported after #2035, but neither deactivated nor settled. - The same loss happens when the gateway itself is down for two epochs: no settlement runs without it.
Reproduce
- Run a gateway with one active escrow that has served requests.
- Damage the escrow's SQLite journal so that recovery reports
ErrLocalStateUnrecoverable, and restart the gateway. - The escrow is deactivated and excluded from routing. Nothing settles it. Two epochs later the chain splits the deposit per slot.
Expected: an escrow that cannot be served is still settled before its settlement window closes, with the hosts' real stats.
🔄 Auto-synced from Issue #2039 every hour.