Skip to content

devshard: host bridge drops escrow refusal_timeout/execution_timeout, so host and gateway can bind different SessionConfig #1762

Open @kAIPraxisBot opened 2026-09-13 03:44 UTC 6 comments Updated 2026-09-18 20:26 UTC

On devshard-0.2.15-v5 (PR #1584), the gateway and the host build SessionConfig with the same function but fetch the escrow through different bridges. The gateway's bridge copies refusal_timeout / execution_timeout off the escrow row; the host's bridge does not, so the host silently falls back to the compiled defaults (60 / 1920). Both values feed consensus — auto-seal and timeout votes — so a governance value other than the default desynchronises the two sides.

All line references are at PR head 9d1f01472459f17b1516933951d2617027b0ad08.

Where the two paths diverge

The gateway uses bridge.GRPCBridge (main.go:527, gateway.go:410), which carries both fields:

// devshard/bridge/grpc.go:99-104
        ValidationRate:            e.ValidationRate,
        VoteThresholdFactor:       e.VoteThresholdFactor,
        RefusalTimeout:            e.RefusalTimeout,
        ExecutionTimeout:          e.ExecutionTimeout,
        EpochID:                   e.EpochIndex,
        Settled:                   e.Settled,

The host (devshardd) uses CachingEscrowBridge over ChainBridge (app.go:276), and that bridge's EscrowInfo literal ends without them:

// devshard/cmd/devshardd/bridge/chain.go:130-137
        InferenceSealGraceNonces:  e.InferenceSealGraceNonces,
        InferenceSealGraceSeconds: e.InferenceSealGraceSeconds,
        AutoSealEveryNNonces:      e.AutoSealEveryNNonces,
        ValidationRate:            e.ValidationRate,
        VoteThresholdFactor:       e.VoteThresholdFactor,
        EpochID:                   e.EpochIndex,
        Settled:                   e.Settled,
    }, nil

The warm cache cannot restore them either — EscrowCacheInfo has no such fields, so EscrowInfoFromCache maps a cached row back without them as well.

Both sides then go through the same mapper, SessionConfigAtBind → SessionConfigFromEscrow, where zero means "use the compiled default":

// devshard/types/config.go:50-53
func DefaultSessionConfig(groupSize int) SessionConfig {
    return NormalizeSessionConfig(SessionConfig{
        RefusalTimeout:    60,
        ExecutionTimeout:  32 * 60,

Why both values are consensus-relevant

Auto-seal reads ExecutionTimeout from the session config (seal.go:452) and folds it into the Finished clock gate:

// devshard/state/seal.go:43-51
func FinishedClockRequiredSeconds(graceSeconds, executionTimeout int64) int64 {
    ...
    return graceSeconds + executionTimeout
}

autoSealLocked's own comment states the consequence: "Mixed binaries that disagree on this sum diverge on SealedAcc / post_state_root."

Host-side timeout votes use the same config — timeout.go:109 rejects a REFUSED vote while now - StartedAt < RefusalTimeout, and timeout.go:183 rejects an EXECUTION vote while now - ConfirmedAt < ExecutionTimeout — while the gateway computes its own deadline from its own config (session.go:2824).

Reproduction

Same mapper, same group size, one field differing: the host's escrow (fields dropped by ChainBridge) against the gateway's (fields carried by GRPCBridge) for a governance execution_timeout = 1200, with inference_seal_grace_seconds = 3600:

hostCfg := SessionConfigAtBind(16, &hostEscrow)        // RefusalTimeout/ExecutionTimeout zero
gatewayCfg := SessionConfigAtBind(16, &gatewayEscrow)  // RefusalTimeout 60, ExecutionTimeout 1200
state.FinishedClockRequiredSeconds(int64(cfg.InferenceSealGraceSeconds), cfg.ExecutionTimeout)

SUBJECT host    (ChainBridge drops the fields): refusal=60 execution=1920 sealThreshold=5520
CONTROL gateway (GRPCBridge carries them):      refusal=60 execution=1200 sealThreshold=4800

Two consequences follow from that gap:

  • Sealing. For an inference whose stateClock - ConfirmedAt lands between 4800 and 5520, the gateway folds it into SealedAcc and the hosts do not, so post_state_root disagrees. Hosts answer post_state_root does not match computed state root (types/errors.go:35) and the gateway marks them escrow_state_root_diverged (redundancy.go:3787). A raised timeout produces the mirror image.
  • Votes. The gateway starts voting at its own (smaller) deadline while hosts still reject the vote as premature, so quorum is never reached and the record stays Started. At settlement a still-live record pays the executor its full reservation (machine.go:1668):
    case types.StatusStarted, types.StatusPending:
        rec.ActualCost = rec.ReservedCost
        rec.Status = types.StatusFinished
        sm.state.HostStats[rec.ExecutorSlot].Cost += rec.ReservedCost

Why it is quiet today

Mainnet governance currently sets exactly the compiled defaults, so the two paths agree by coincidence. Read from the public REST endpoint (/chain-api/productscience/inference/inference/params on node1.gonka.ai:8000), 2026-09-13:

devshard_escrow_params.refusal_timeout = 60
devshard_escrow_params.execution_timeout = 1920
devshard_escrow_params.default_inference_seal_grace_seconds = 3600

Nothing detects the mismatch on its own: SessionConfig is not part of the state root (hash.go:38 folds balance, host stats, inferences, phase, warm keys, fees and version), so a config divergence only ever surfaces later as a root mismatch or a stuck vote.

Stands do change these values through governance — RuntimeConfigTests.kt bumps refusal_timeout and execution_timeout — so a non-default value is reachable in test environments before it is ever reachable in production.

Origin

Commit 88ebd4456 (#1564) added refusal_timeout = 17 / execution_timeout = 18 to inference/inference/devshard_escrow.proto together with the gRPC-bridge mapping and SessionConfigAtBind, and touched neither devshard/cmd/devshardd/bridge/chain.go nor devshard/storage/interface.go. On the branch's merge base 379bebced638aeb5e6077bfd51c986f898443832 none of the three files mentions either field.

Suggested fix

  1. Copy both fields in ChainBridge.GetEscrow (devshard/cmd/devshardd/bridge/chain.go:120), mirroring GRPCBridge.
  2. Add both to storage.EscrowCacheInfo and to both cache mappers (EscrowCacheFromInfo / EscrowInfoFromCache) — without this half, a bind served from the warm cache still loses them.
  3. Consider a test that feeds one bridge's real EscrowInfo into the other's bind path and asserts field-for-field equality, since each side's tests build their own fixtures today.
  4. Two related points worth deciding separately:
  5. devshard/user/httpsession.go:156 lets gateway-side configuration override both timeouts after the bind mapping, so the two sides can still be desynchronised by a flag once the bridge is fixed.
  6. devshard/docs/params-dataflow.md is now stale on this: the lane A table (line 27) does not list the two fields, line 45 still files them under lane C as devshardctl-only, and line 52 says SessionConfig carries the defaults.

Not checked

  • No end-to-end run on a stand: the sealing divergence above is code reading plus arithmetic, not an observed root mismatch.
  • Whether any escrow row already on mainnet carries non-zero values in fields 17/18 — the chain release carrying this proto is not what mainnet runs today.

💬 Comments (6)

@kAIPraxisBot commented 2026-09-13 03:45 UTC

@tcharchian yo, mind sanity-checking this one? tl;dr: the gateway's bridge maps refusal_timeout/execution_timeout off the escrow row, devshardd's ChainBridge doesn't, so the host silently falls back to the compiled 60/1920 while the gateway runs the real governance values.

No-op on mainnet today purely because governance happens to sit exactly on the defaults — but bump execution_timeout to 1200 on a stand and the Finished clock gate goes 4800 (gw) vs 5520 (hosts), which is post_state_root mismatch territory plus timeout votes that never reach quorum. Repro output and line refs are in the body.

Fix looks like a two-liner in chain.go, but the warm cache is the other half (EscrowCacheInfo has no such columns) — would be great if you could eyeball whether that needs a schema bump/migration or just the mappers. Also flagged two adjacent things in there: the gateway-side override in httpsession.go and the now-stale lane tables in params-dataflow.md.

@aikuznetsov commented 2026-09-13 14:24 UTC

Validated on current devshard-0.2.15-v5 HEAD (a8b5c00c). The issue is reproducible: both ChainBridge and the warm-cache path drop refusal_timeout / execution_timeout, causing the host to fall back to 60/1920 while the gateway keeps the escrow values. A minimal test with execution_timeout=1200produced seal thresholds of 5520 on the host versus 4800 on the gateway. Mainnet currently uses the compiled defaults, so the bug is real but masked there.

@redstartechno commented 2026-09-14 20:32 UTC

I'll take this one: add RefusalTimeout/ExecutionTimeout to the ChainBridge escrow mapping and the cached path in devshardd, with a regression test that pins both bridges to the same SessionConfig. PR against devshard-0.2.15-v5 shortly.

@a-kuprin commented 2026-09-16 14:32 UTC

Extremely important fix, as it can lead to state divergence in v5 (fires in integration testermint tests) Added to v5 release as a musthave fix: https://github.com/gonka-ai/gonka/pull/1584/changes/3d1b445e17a672d3b2f123a26ced688fff5bd791

@qdanik commented 2026-09-18 19:29 UTC

@a-kuprin can we close this one?

@a-kuprin commented 2026-09-18 20:26 UTC

Let's merge v5 to main and close then

🔄 Auto-synced from Issue #1762 every hour.