Skip to content

devshard gateway: inter-chunk timeout logs stalled SSE streams but does not cancel them #1798

Open @aikuznetsov opened 2026-09-18 02:37 UTC 0 comments Updated 2026-09-18 21:23 UTC

A successful HTTP status only means that the response headers were received. The request can still hang indefinitely while reading the SSE body if the upstream keeps the connection open without sending another event.

This can happen here:

https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/transport/client.go#L333-L349

After detecting text/event-stream, the client enters parseSSEResponse(), which blocks while waiting for the next SSE line:

https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/transport/client.go#L381-L420

If the connection remains open but no additional bytes, EOF, or socket error arrive, readBoundedSSELine() does not return. As a result, no error reaches the retry logic, so no retry is started.

Existing timeout behavior

The gateway defines an inter-chunk timeout and describes it as the limit after which a stalled winner should be aborted:

https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L341-L354

However, the configured InterChunkStallTimeout is currently not used when calculating the stall deadline. The code uses the separate InterChunkStallLogThreshold instead:

https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L2114-L2132

When that timer fires, the gateway only records winner_stalled_after_content. It does not cancel the request:

https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L2430-L2463

The request is actually cancelled only when StreamingAttemptHardTimeout fires:

https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L2464-L2483

That timeout defaults to 30 minutes:

https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L38-L41

Expected behavior

Once the winning stream has produced content, receiving no meaningful SSE events for InterChunkStallTimeout should:

  1. Cancel the upstream request.
  2. Close the response body.
  3. Return a typed retryable error.
  4. Retry against another eligible host, if possible.

The timeout should track meaningful data: events rather than arbitrary network activity. SSE comments such as : keepalive should not keep a generation alive indefinitely.

Suggested change

Use InterChunkStallTimeout as a sliding deadline based on the timestamp of the last meaningful SSE event. When the deadline expires, call the attempt's cancellation function and classify the result as a retryable stalled-stream failure.

The existing 30-minute StreamingAttemptHardTimeout should remain as an absolute safety limit, independent of stream activity.

🔄 Auto-synced from Issue #1798 every hour.