devshard: store I/O under the per-escrow Session.mu serialises all requests of an escrow under load #1986

Open @kaileido opened 2026-10-12 22:21 UTC 0 comments Updated 2026-10-12 22:22 UTC

Session.mu (devshard/user/session.go:240) is one lock per escrow session, and the gateway runs one session per escrow, so it serialises every request on that escrow. Durable store I/O runs while it is held:

  • ProcessResponse writes store.AddSignature once per validator slot (session.go:964-968).
  • PrepareInferenceFn persists the diff through persistDiffRetryLocked → storage.AppendDiffWithRetry, with backoff retries, under the lock (session.go:1319-1332). This one is deliberate: releasing the lock during backoff would let a concurrent compose sign the same next nonce.
  • For a host that lags more than the in-memory diff suffix, diffRangeLocked (:685) and postStateRootForNonce (:870) read diffs from the store under the lock.

Under load every lock hold carries a store round trip, so lock wait grows with request rate and with each speculative secondary attempt, which adds another PrepareInferenceFn. It shows up as growing session-mutex wait in the long-escrow 32 rps load scenario.

Proposal: keep AppendDiff under the lock, since it guards against a diff fork. Move AddSignature out of it: update s.signatures under the lock and persist the signature after unlock or through an ordered background writer. A failed signature write is already only logged (failed to persist signature), so durability semantics do not change. Optionally, read the lagging-host diff range outside the lock and re-check the cursor under it.

To verify: compare gateway mutex profiles from devshard/testenv/loadtest long-escrow-32rps-8m before and after.

🔄 Auto-synced from Issue #1986 every hour.