Skip to content

[BUG] Gateway/mlnode: agent-sized prefill hangs or fails with no usable error #1677

Closed @paranjko opened 2026-08-29 20:39 UTC 1 comment Updated 2026-08-29 21:51 UTC

Summary

Chat completions whose total prefill is on the order of ≥ ~8K tokens (typical agent/RAG system prompts; not a single-message format bug) often never produce a first token: client sees a hang with 0 bytes, a write timeout, or a generic upstream failure. Small prompts on the same model/path return HTTP 200 in ~1s. Probaly a gateway → host/vLLM prefill problem.

Split out of https://github.com/gonka-ai/gonka/issues/1628.

Motivation

The network advertises 200K–400K context. Agent clients in docs/chat-api/agents.md routinely send tens of thousands of input tokens. If that class of request hangs or takes the host down, the advertised developer use case does not work. Repeated fails also feed host quarantine.

What to reproduce

  1. Same model, same router: messages=[{user: "ping"}], max_tokens=5, stream=true → HTTP 200, first token < 2s.
  2. Same call with total input ~8K–40K tokens (one message or split across many — #1628 showed chunking does not help) → hang > tens of seconds with 0 bytes, client abort, or opaque 5xx.
  3. Confirm it is prefill volume, not JSON shape.

💬 Comments (1)

@paranjko commented 2026-08-29 21:51 UTC

Could not reproduce a 0-byte hang on our gateway (DeepSeek-V4-Flash-0731 and MiniMax-M2.7, unique ~12k / ~63k prompt_tokens). stream=true and stream=false: HTTP 200. Stream: first token ~1–6s. Non-stream: full JSON in ~2–15s (DeepSeek ~63k was 14.5s). max_tokens=5 completes as 64. Engine stayed up.

A ~75k-in / 4096-out MiniMax non-stream on another path finished in ~53s (outcome=served) a bit slow, not stuck.

Asked the topic starters on #1628 for the large-prompt curl.

Probably not a network-wide ≥8K 0-byte hang. Closing here.

🔄 Auto-synced from Issue #1677 every hour.