[BUG] Public routers hang/timeout on prompts ≥ ~7.5K tokens (prefill); 502 `all_providers_failed`; DeepSeek missing from /v1/models #1628

Open @inecro1 opened 2026-08-23 12:28 UTC 7 comments Updated 2026-09-12 14:29 UTC

Summary

Public Gonka routers hang or fail on any request whose total prefill is ≥ ~7.5K tokens: tested 2026-08-23 across api.opengonka.com, node.gonka.lat, gate.joingonka.ai, openbroker.gonka.gg (and partially proxy.gonka.gg). Small prompts (5 tokens) return HTTP 200 in ~1 s; prompts at ~40K tokens hang >90 s with zero bytes, or fail immediately with HTTP 500/502 (all_providers_failed). The same failure reproduces when the same total volume is split across 10 messages — it is a prefill-volume problem, not a single-message size/format problem. Separately, the /v1/models catalog is inaccurate: active deepseek-ai/DeepSeek-V4-Flash-0731 answers direct requests but is absent from the catalog, while retired Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 is listed and always fails.

Motivation

The network advertises 200K–400K context models (DeepSeek-V4-Flash-0731: 380–400K per node-config, proposal 94). LLM agents (e.g. Hermes Agent, which is in the official compatibility matrix in docs/chat-api/agents.md) routinely send system prompts of tens of thousands of tokens. At the current prefill threshold the network is unusable for its advertised primary use case (agent workloads), not just for adversarial large inputs. The catalog inaccuracy compounds this by making the active PoC model unselectable from web UIs while retired models mislead clients into guaranteed failures.

Impact

Who is affected (hosts, developers, validators):
End users / developers via public routers (api.opengonka.com, node.gonka.lat, gate.joingonka.ai, openbroker.gonka.gg, proxy.gonka.gg): any agent/RAG client with a large system prompt cannot complete a single request.
Gateway operators: repeated all_providers_failed paths trigger mass host quarantine cycles (see #1506), degrading health scoring for honest hosts.
Web chat users: 502/429/402 storms on simple questions (observed on opengonka web chat and browser/network logs).
Is effect network-wide or limited: Network-wide pattern — reproduced independently on 4 public routers (including the official gate.joingonka.ai), not a single-host fault. Matches documented vLLM OOM behavior (#1171) rather than one operator's config.
Likelihood (common, intermittent, edge case, or intentional attack): Common — deterministic at ≥ ~7.5K prefill tokens on the tested routers; not an edge case.
Severity [Impact x Likelihood]: High — common × network-wide impact for the advertised agent/developer use case (see risk matrix in FAQ).
Affected components: public gateway/router layer (proxy → gateway → mlnode/vLLM prefill), /v1/models catalog serving, gateway host-health/quarantine logic (#1506 interplay).

Detailed description

Reproduction (all tests 2026-08-23)

Environment: OpenAI-compatible client (curl and Hermes Agent custom_providers + model_aliases, api_mode: openai); API key issued by opengonka.com (gnk-sk-…); account balance sufficient (10M test tokens + 77 GNK — failures are NOT client billing).

Small prompt — works, ~1 s:

curl -N -sS https://api.opengonka.com/v1/chat/completions \
  -H "Authorization: Bearer $GONKA_ROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-ai/DeepSeek-V4-Flash-0731","messages":[{"role":"user","content":"ping"}],"max_tokens":5,"stream":true}'
# → HTTP 200, first token < 2 s

Large prompt (~40K tokens) — fails on every router:

Router	Result
`https://api.opengonka.com/v1`	timeout >90 s, 0 bytes (client `--max-time 90` aborted)
`https://node.gonka.lat/v1`	HTTP 502 `{"error":{"message":"All providers failed to respond","type":"upstream_error","code":"all_providers_failed"}}`
`https://gate.joingonka.ai/v1`	write timeout (client-side, stream never opened)
`https://openbroker.gonka.gg/v1`	404 on `/v1/models` (broker path unavailable)

Chunking does not help: the same ~40K token total split into 10 messages still hangs (>90 s, 0 bytes). The trigger is total prefill volume, not single-message size or message format.

Threshold: failure is deterministic at ~7.5K prefill tokens and above. This is consistent with the documented vLLM v1 OOM at ~6K+ tokens with forced logprobs (issue #1171: EngineDeadError → HTTP 500, engine down 6–12 min) and with the gateway forcing logprobs=true, top_logprobs=5, return_token_ids=true for observability (docs/chat-api/README.md).

Catalog inaccuracy

GET /v1/models (api.opengonka.com, node.gonka.lat, gate.joingonka.ai): Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 (retired by proposal 78) is listed; every request to it returns 502 all_providers_failed.
deepseek-ai/DeepSeek-V4-Flash-0731 (active PoC model, epoch 360, proposal 94) answers direct requests with HTTP 200 but is absent from /v1/models — web UIs cannot select it.

Notes on evidence and logs

Server-side router/gateway logs are not accessible to the client; the client-side evidence is exact HTTP codes, timing, and request behavior above. We can provide x-request-id values from api.opengonka.com responses on request (the router returns x-request-id and X-Provider headers on every response).
Prompt caching is NOT available: cache_key / prompt_cache_key are silently stripped by the gateway (docs/chat-api troubleshooting), so every large request is a cold prefill.
Related open issues: #1171 (vLLM OOM on ~6K+ prompts), #1550 (DRAFT session affinity / KV-cache reuse), #1591 (request continues after client timeout), #1506 (mass quarantine → no winners), #1121 (429 storms, p50 TTFT 19.7 s on gate.joingonka.ai), #1579/#1574 (always-stream upstream), #424 (reliability math).

Suggested actions

Publish an accurate /v1/models: remove retired Qwen3-235B, add active deepseek-ai/DeepSeek-V4-Flash-0731 (and any other active models).
Fix prefill/OOM on long prompts: memory handling in mlnode/vLLM, stop forcing logprobs on every request, KV-cache reuse (see #1550).
Make gateway timeouts surface server-side errors instead of silent 0-byte hangs or all_providers_failed (correlate with #1591/#1593).
Stabilize public routers against 429/502 storms (#1121, #1506).
Consider surfacing real-time router health (gonka.pw-style) in the official quickstart so users can pick a healthy gateway.

💬 Comments (7)

@theaungmyatmoe commented 2026-08-23 17:02 UTC

It's the widely occurred bug it need attention

@paranjko commented 2026-08-29 21:39 UTC

Tried several times the ping-vs-large-prefill contrast from this thread on our gateway: DeepSeek-V4-Flash-0731 and MiniMax-M2.7, stream=true, unique padding.

Both models: HTTP 200, first token in ~1–6s at ~12k and ~63k prompt_tokens. max_tokens=5 completes as 64 (gateway floor). No 0-byte hang; engine not left dead.

A ~75k-in / 4096-out MiniMax non-stream on another path finished in ~53s (outcome=served) a bit slow, not stuck.

The curl in the post is the ping. To replay the hang, please paste the same kind of command with the large prompt (~7.5K or ~40K) that timed out for you (URL + body + --max-time).

Happy to re-run if we have that. Gonka External TestLab

@inecro1 commented 2026-08-30 10:51 UTC

Thanks for the detailed test! One clarification: our report is specifically about the public routers (api.opengonka.com, node.gonka.lat, gate.joingonka.ai, openbroker.gonka.gg) — your gateway is a separate deployment, so a clean pass there doesn't contradict the report.

Here's the exact repro we use (large-prefill non-stream request to api.opengonka.com, ~160K chars ≈ 40K tokens system prompt; redact the key):

python3 - <<'EOF'
import json, urllib.request, time
body = json.dumps({
  "model": "deepseek-ai/DeepSeek-V4-Flash-0731",
  "messages": [
    {"role": "system", "content": "x" * 160_000},
    {"role": "user", "content": "ok"}
  ],
  "max_tokens": 5,
  "stream": False,
}).encode()
req = urllib.request.Request("https://api.opengonka.com/v1/chat/completions", data=body, method="POST")
req.add_header("Authorization", "Bearer <API_KEY>")
req.add_header("Content-Type", "application/json")
t0 = time.time()
try:
    with urllib.request.urlopen(req, timeout=100) as r:
        print("OK", round(time.time()-t0, 1), "s", r.read()[:200])
except Exception as e:
    print(type(e).__name__, round(time.time()-t0, 1), "s", str(e)[:200])
EOF

Latest probe results against api.opengonka.com (2026-08-30): - GET /v1/models -> 200 in ~0.5s - small prompt (~5 tokens) -> 200 in 1.2-3.4s - same large-prefill request as above -> timeout after 100s, zero bytes (reproducible on every probe since 2026-08-23, latest at 13:14 today)

We also reproduced the failure across node.gonka.lat, gate.joingonka.ai, openbroker.gonka.gg, and with stream=true (first chunk never arrives). If it works on your gateway, great — the question is why the public routers still hang at the same prefill volume. Happy to run a side-by-side test against your gateway if that helps.

@gonkalabs commented 2026-09-03 22:38 UTC

Hi @inecro1 - thanks for the clarification and the exact repro.

There are two claims in the issue: (1) /v1/models is wrong, and (2) public routers hang or return all_providers_failed on prefill ≥ ~7.5K.

We operate proxy.gonka.gg and OpenBroker and response will be only around them (also: we can not answer on behalf of brokers that are operating using OpenBroker as a gateway because we can not be sure on how request/response transformation flow might happen there)

After your 2026-08-30 note we re-checked both of our live paths. We do not see that hang signature.

proxy.gonka.gg

Catalog. This is the opposite of “retired Qwen listed, DeepSeek hidden.” Live GET /v1/models on both https://api.proxy.gonka.gg/v1/models and https://proxy.gonka.gg/v1/models returns:

deepseek-ai/DeepSeek-V4-Flash-0731
MiniMaxAI/MiniMax-M2.7
moonshotai/Kimi-K2.6

Qwen is not in the catalog. Old clients that still send Qwen get 404 not available, not a fake “it’s up” 502.

Preferred base URL: https://api.proxy.gonka.gg/v1 (docs). The apex host also serves /v1 today, but clients should follow the docs as proxy.gonka.gg/v1 is a frontend-re-resolved route, not direct production api path (it is reserved for reverse-compatibility for old clients, but it is limited according to frontend operation traffic rules).

Large prefill / all_providers_failed all_providers_failed. We do not see “≥ ~7.5K and the engine dies / 0-byte hang” on the current proxy path. Large agent-sized prefills complete, including DeepSeek prompts well above 40K. We have zero all_providers_failed rows on this router.

What we do see is the other half of this thread: capacity, not a hard prefill cliff - 429 (too many concurrent), 503 (hosts / queue full), 502 (no host left / deadline). That is the #1121 / #1506 family. It is network state, not “large prompts never work,” and not broker-specific.

If proxy looked “partially” affected on Aug 23, that specific 0-byte / all_providers_failed prefill cliff is not what we see on the current path now.

OpenBroker

/v1/models /v1/models. Your original table already has this right: https://openbroker.gonka.gg/v1 → 404. That host is the website, not the API.

The API is https://api.openbroker.gonka.gg/v1. GET /v1/models returns the same three live models (no Qwen). It is documented at openbroker.gonka.gg/docs (GET /models) and only this api path shall be used.

Large prefill. We ran your exact script (DeepSeek, "x" * 160_000, max_tokens=5, 100s timeout) against https://api.openbroker.gonka.gg at 2026-09-03 22:27 UTC:

Probe	Result
small prompt	200 in 0.3s
your large non-stream body	200 in 0.9s (~20k prompt tokens - repeated `x` compresses; not ~40k)
same body, `stream=true`	200 in 1.8s, first chunk arrived
same calls on `openbroker.gonka.gg`	404 immediately (website / frontend resolved path and it is expected. Api path is api.openbroker.gonka.gg as per the docs).

Last 24h on OpenBroker: 130k+ requests with prompt ≥ 7.5K and 14.5k+ with ≥ 40K, all HTTP 200. Max logged prompt ~227K. Zero all_providers_failed (that error is from other routers). We do not force logprobs on chat.

Failed rows often have prompt_tokens=0, so a true silent hang would not land in the ≥7.5K bucket. What we can say is: agent-sized prefills are completing here now, and the OpenBroker line in the issue is mostly the website host 404 / wrong base URL, not the 0-byte hang.

Our 502s here are mostly client cancel (if code that calls our api from client side does have a very short timeout value) and “no host left” (capacity), not a prefill OOM.

If you still see a hang, please re-run that script against https://api.openbroker.gonka.gg/v1/chat/completions and https://api.proxy.gonka.gg/v1/chat/completions, and send UTC time + x-request-id. Happy to do a side-by-side (same applied for proxy.gonka.gg issues with correct api.proxy.gonka.gg path as per docs).

Thank You for the report,

Gonka Labs team

@inecro1 commented 2026-09-06 09:40 UTC

Hi @gonkalabs — follow-up from our side after your 2026-09-03 reply. We ran a side-by-side probe on 2026-09-06 09:14 UTC (same key, same model deepseek-ai/DeepSeek-V4-Flash-0731, stream=true/false as applicable) against all three paths.

api.opengonka.com/v1 (the router our key is issued for)

GET /v1/models -> 200 (~0.6s). Catalog now: MiniMaxAI/MiniMax-M2.7, deepseek-ai/DeepSeek-V4-Flash-0731, moonshotai/Kimi-K2.6, zai-org/GLM-5.2-FP8. Qwen is gone and DeepSeek is listed — the catalog part of the issue appears resolved on this router. ✅
small prompt (stream) -> 200 in 2.2s
~7.7K prefill (stream) -> timeout after 60s, zero bytes (no x-request-id in headers — they never arrived)
~40K prefill ("x"*160_000 system, non-stream, max_tokens=5, your exact repro) -> timeout after 110s (read operation timed out)
~40K prefill (stream) -> timeout after 110s

So the ≥~7.5K prefill hang is still 100% reproducible on api.opengonka.com as of today.

api.proxy.gonka.gg/v1 and api.openbroker.gonka.gg/v1 (your paths)

GET /v1/models -> 200 on both (~0.4s; catalog on both: DeepSeek-V4-Flash-0731 + MiniMax-M2.7)
chat/completions with our opengonka-issued gnk-sk-… key -> 401 Unauthorized (~0.2s) — the key is not accepted on your paths
~7.7K and ~40K prefills with the same key -> do NOT return 401 quickly: they hang 60–110s (read/write operation timed out) before any auth response

What this means / request

We cannot verify your "no hang on our paths" claim from this account: the routers use separate billing/key systems, and our key gets 401 on proxy.gonka.gg / OpenBroker. Two questions: 1. Where do we obtain a key for api.proxy.gonka.gg / api.openbroker.gonka.gg (registration hub / docs page)? We'd like to run the exact repro with valid auth on your paths and close this out. 2. Is the pre-auth hang on large bodies (60–110s instead of a fast 401) expected behavior on your gateways — i.e. is the full request body read/processed before authentication? If so, that still looks like a prefill-path problem worth a look.

Happy to provide UTC times and full request bodies for any specific run you want to inspect.

@gonkalabs commented 2026-09-11 23:53 UTC

Hi @inecro1 !

If you were able to use openbroker.gonka.gg / proxy.gonka.gg - api keys are the same for api.openbroker.gonka.gg and proxy.gonka.gg.
Gonka Labs has nothing to do with api.opengonka.com/v1, this is not our product and api keys of that service will not work with *.gonka.gg services.
There is no hang on large bodies on *.gonka.gg services: Open Broker and Proxy. The hang issue shall be discussed with opengonka maintainers.

Thank You for the report,

Gonka Labs team

@inecro1 commented 2026-09-12 14:29 UTC

Follow-up: consumer Proxy path also fails above ~4K–10K prefill; consumer key is rejected on OpenBroker

Hi @gonkalabs — we followed your guidance literally. On 2026-09-12 we signed up at proxy.gonka.gg (email + verification code, no KYC), issued a key from the dashboard, and ran the same prefill contrast through https://api.proxy.gonka.gg/v1. Model deepseek-ai/DeepSeek-V4-Flash-0731 (and MiniMaxAI/MiniMax-M2.7 for the largest attempt), single user message with repeated filler, stream=false, max_tokens=5, ~13:50–14:10 UTC.

prompt_tokens	result
39	200, 0.2 s
1,775	200, 0.5 s
2,005	200, 0.8 s
4,005	200, 4.3 s
~10,000	no bytes, 40 s (same with `stream=true`)
~40,000	no bytes, 70 s
~96,000	connection dropped at 120 s — identical for MiniMax-M2.7

So on the Proxy path the ceiling on our account sits between ~4K and ~10K prefill tokens. Same failure signature as the original report, on a path you described as having no hang on large bodies.

For scale: your own /api/status taken at the same moment shows 452,436 requests and 2,535,829,975 tokens over the last 24 h — an average of ~5,600 tokens per request, i.e. below the ceiling we hit. Our client is an agent that sends 96K–111K prompt tokens per call (measured: 632M input tokens over 5,668 calls), so Proxy is currently unusable for us.

To rule out a network-side cause: the identical ~96K body returns 200 OK in 2.0 s (usage.prompt_tokens = 96,005) through api.gonka24.com, so the devshard path is capable of serving this size.

One contradiction to clarify: our proxy.gonka.gg key is rejected on your other endpoint — POST https://api.openbroker.gonka.gg/v1/chat/completions → 401 {"error":"missing or invalid OpenBroker API key"} — although you said "api keys are the same for api.openbroker.gonka.gg and proxy.gonka.gg".

Questions

Is there a request-size cap on the consumer/trial tier of Proxy, and does it lift after a top-up? What is the maximum prompt a paying consumer account can send?
If Proxy is not intended for 100K+ prompts, which product/tier is (OpenBroker broker account with reserved devshard capacity?), and what would that cost for a single-user agent workload at ~640M tokens/month?
Should a proxy.gonka.gg key work on api.openbroker.gonka.gg, or do OpenBroker keys come only with the ≥100 GNK broker activation?
If useful, we can share exact UTC timestamps and request bodies so you can check in your logs whether our ~10K/40K/96K requests reached a devshard at all.

Happy to re-run the exact repro on any path you specify with a key that works there.

🔄 Auto-synced from Issue #1628 every hour.