Skip to content

Gateway host-health deadlock: mass quarantine → all hosts stuck as no-winner/suspicious #1506

Open @maria-mitina opened 2026-07-27 09:48 UTC 3 comments Updated 2026-07-27 10:27 UTC

Summary

I managed to get all hosts into a kind of deadlock on testnet.

How everyone entered quarantine

I created an escrow with the wrong owner (CLI key ≠ DEVSHARD_PRIVATE_KEY the gateway signs with), registered it on the gateway, and sent chat. The gateway broadcast / raced the request across participants. Every host correctly rejected with HTTP 403 (restricted to escrow owner). The gateway treated each 403 as a host health signal and applied a ~30 min inference quarantine to each host that answered.

I understand there are multiple ways to enter quarantine (404 session missing, transport failures, empty streams, etc.). On a small testnet this was easy to trip for the whole set at once; on mainnet that mass fan-out should be harder, but the same mechanism still exists.

How they got stuck after unquarantine

After I cleared quarantine via POST /v1/admin/participants/unquarantine, every host moved into suspicious / no-winner probation (failure_strikes left after quarantine; attempts marked primary_no_winner / suspicious_winner_deferred).

They never got out, because there were no non-suspicious winners left who could serve the client. Probation only clears when finished good attempts call ObserveSuccessfulInference (strike decrement). With every usable host on probation:

  1. Hosts still run inference and return real content
  2. Content is deferred (suspicious_winner_deferred, winner_nonce=0) — not crowned to the client
  3. Request fails / times out
  4. Strike recovery does not progress in practice when no clean host can complete the request for the client

So: wrong-owner 403 → mass quarantine → mass unquarantine into probation → no clean crown path → permanent suspicious deadlock.

Observed on

  • Host: testnet gateway .79
  • Image: ghcr.io/gonka-ai/devshard-gateway:testnet-v4-local (built from devshard-0.2.14-v4)
  • Log signals: participant_limit_activated status=403, decision=primary_suspicious, primary_no_winner_reason=probation, suspicious_winner_deferred, picker_exhausted

Suggested product fixes

  1. Do not quarantine participants on 403 “restricted to escrow owner” (misconfigured escrow / signer mismatch, not a bad host).
  2. When no clean host remains, crown a suspicious winner that already returned good content (or accept deferred completions toward strike recovery).
  3. Add an admin “forget / fully clear” path that lifts quarantine without leaving the host in no-winner probation (separate from soft unquarantine).

Ops workaround that clears the stuck state

Restart the gateway after throttle rows are cleared (in-memory probation dies with the process). Soft unquarantine alone is not enough — it re-enters probation by design.

Other ways to reach the same mass-quarantine shape

Same pattern every time: one shared gateway/request fault × redundancy race × immediate quarantine = whole participant set dirty. Then unquarantine (or natural expiry) → everyone on no-winner probation → no clean crown path.

Immediate mass quarantine (one bad chat can hit everyone)

Trigger Duration Shared cause that fans out
403 Forbidden on /chat/completions ~30m Wrong escrow owner (this incident); other host auth rejects that return 403 on inference
404 Not Found on inference ~30m Dead/missing session on hosts (session not found), escrow never registered, hosts wiped session storage while the gateway still routes it
401 + timestamp drift ~30m Gateway clock skew vs hosts, or signed requests aged in a long queue before send (>30s)
429 / 503 ~60m All hosts overloaded (PoC / capacity / load); every raced host reports throttle
Non-EOF transport failure ~30m Gateway cannot reach anyone (bad participant URLs, network cut, TLS/dial failures across the set)

These are the dangerous ones: one attempt per host → quarantine, no strike streak required.

Slower / less “one-shot” mass quarantine

Trigger Needs Mass risk
Empty stream 3 strikes → shadow quarantine Needs repeated empty finishes; empties are only counted when another attempt succeeded — harder in a total failure, easier under partial success + bad model/max_tokens across hosts
EOF transport 3 consecutive EOFs Flaky shared path (proxy, LB) can accumulate under load
Stalled winner Immediate, but only the crowned winner One host at a time — not a true mass event

Vote/gossip transport failures are intentionally ignored and do not quarantine.

Same deadlock after any of the above

Mass quarantine → admin unquarantine (or natural expiry) → all hosts enter no-winner probation (failure_strikes left after quarantine) → if no clean host remains, suspicious_winner_deferred forever.

Highest-likelihood mainnet analogues

  1. 404 / session missing after restart, bad register, or escrow already gone
  2. Timestamp drift during backlog / NTP issues
  3. 429/503 during PoC or overload if the race still fans out
  4. Transport if the gateway’s view of participant endpoints is wrong for everyone

Wrong-owner 403 is the cleanest demo on a small testnet; on mainnet, 404 + drift + overload are the realistic mass paths.

💬 Comments (3)

@maria-mitina commented 2026-07-27 09:52 UTC

Gateway log evidence (testnet .79)

Full excerpt: https://gist.github.com/maria-mitina/1da0f9c348343d406b90483a2c8d5e4d

Image: ghcr.io/gonka-ai/devshard-gateway:testnet-v4-local
When: 2026-07-27 ~08:55–09:12 UTC

A) Wrong-owner escrow 23 → every host 403 → mass quarantine

After admin clear, chat against escrow 23 races all hosts; each returns restricted to escrow owner and is quarantined:

2026/07/27 08:55:47 participant_quarantine_cleared ...zena4439
2026/07/27 08:55:47 decision_made escrow=23 ... decision=primary_suspicious primary_no_winner_reason=probation primary_failure_strikes=2
2026/07/27 08:55:47 participant_limit_activated ...y5fs9j0r status=403 path_kind=inference
2026/07/27 08:55:47 send_failed escrow=23 ... error="... status 403: {\"message\":\"restricted to escrow owner\"}"
... (same 403 + participant_limit_activated for every host in the race) ...
2026/07/27 08:55:50 participant_limit_activated ...ag64jcz4 status=403 path_kind=inference

B) Unquarantine again → still probation / no-winner on escrow 24

2026/07/27 09:00:21 participant_quarantine_cleared ... (all hosts)
2026/07/27 09:00:21 decision_made escrow=24 ... decision=primary_suspicious primary_no_winner_reason=probation primary_failure_strikes=2
2026/07/27 09:00:22 suspicious_winner_deferred escrow=24 ... host=ag64jcz4
2026/07/27 09:00:23 suspicious_winner_deferred escrow=24 ... host=zena4439
2026/07/27 09:00:23 picker_exhausted ... attempt_failed

C) Hosts finish real content but nothing is crowned (client timeout)

Representative request req-...-96 — all 6 escrow hosts send_completed with content, all suspicious_winner_deferred, winner_nonce=0:

2026/07/27 09:12:17 decision_made escrow=24 ... primary_suspicious primary_no_winner_reason=probation
2026/07/27 09:12:17 suspicious_winner_deferred ... y5fs9j0r / xsz307ef
2026/07/27 09:12:18 suspicious_winner_deferred ... zena4439 / vq8mg6gt
2026/07/27 09:12:19 suspicious_winner_deferred ... ag64jcz4
2026/07/27 09:12:22 suspicious_winner_deferred ... 9umz0l4g
2026/07/27 09:12:22 send_completed ... content_chunks=1 (each host)
2026/07/27 09:12:22 picker_exhausted ... tried every host in escrow
@qdanik commented 2026-07-27 10:05 UTC

Q1: should we manage private key verification on the escrow side and prohibit the creation of escrow with an invalid private key?

Q2: should we reset quarantine count after the 30 mins period (now it decrease only 1, for example: 3 -> 2). Quarantine count can be changed through the env variable

Q3: should we reset the escrow quarantines when all participants already marked as quarantined?

@maria-mitina commented 2026-07-27 10:10 UTC

@qdanik as for Q1: i think this may be my testnet trick - at the moment of creation all addresses were allwed to create escrows. this is not the case on Mainnet where we have limited number of allowed brokers, and they will configure the gateway correctly to sign with their private key i guess

🔄 Auto-synced from Issue #1506 every hour.