devshard gateway: inter-chunk timeout logs stalled SSE streams but does not cancel them #1798
A successful HTTP status only means that the response headers were received. The request can still hang indefinitely while reading the SSE body if the upstream keeps the connection open without sending another event.
This can happen here:
https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/transport/client.go#L333-L349
After detecting text/event-stream, the client enters parseSSEResponse(), which blocks while waiting for the next SSE line:
https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/transport/client.go#L381-L420
If the connection remains open but no additional bytes, EOF, or socket error arrive, readBoundedSSELine() does not return. As a result, no error reaches the retry logic, so no retry is started.
Existing timeout behavior
The gateway defines an inter-chunk timeout and describes it as the limit after which a stalled winner should be aborted:
https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L341-L354
However, the configured InterChunkStallTimeout is currently not used when calculating the stall deadline. The code uses the separate InterChunkStallLogThreshold instead:
https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L2114-L2132
When that timer fires, the gateway only records winner_stalled_after_content. It does not cancel the request:
https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L2430-L2463
The request is actually cancelled only when StreamingAttemptHardTimeout fires:
https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L2464-L2483
That timeout defaults to 30 minutes:
https://github.com/gonka-ai/gonka/blob/87940400cd633e514349580185ebce8cfd23502d/devshard/cmd/devshardctl/redundancy.go#L38-L41
Expected behavior
Once the winning stream has produced content, receiving no meaningful SSE events for InterChunkStallTimeout should:
- Cancel the upstream request.
- Close the response body.
- Return a typed retryable error.
- Retry against another eligible host, if possible.
The timeout should track meaningful data: events rather than arbitrary network activity. SSE comments such as : keepalive should not keep a generation alive indefinitely.
Suggested change
Use InterChunkStallTimeout as a sliding deadline based on the timestamp of the last meaningful SSE event. When the deadline expires, call the attempt's cancellation function and classify the result as a retryable stalled-stream failure.
The existing 30-minute StreamingAttemptHardTimeout should remain as an absolute safety limit, independent of stream activity.
💬 Comments (3)
@Ryanchen911 ok, I think #1841 will be in work on this weekends
Implemented in #1917 (base devshard-0.2.x-v6), following the approach above: the 30s/60s pair is now two deadlines on one lastChunkAt anchor, expiry cancels the upstream, and a typed errStreamStalled makes the stall retryable on another host when nothing had reached the client yet.
🔄 Auto-synced from Issue #1798 every hour.
Taking this. The inter-chunk timeout currently only logs but doesn't cancel stalled streams — need to make it actually abort and retry.
Approach: 1. Use
InterChunkStallTimeout(not justLogThreshold) as a sliding deadline based on meaningful SSE events 2. Cancel the upstream request when the deadline expires 3. Return a typed retryable error so redundancy can try another host 4. Keep the 30min hard timeout as an absolute safety limitWill coordinate with ongoing v6 stream-signature work (@a-kuprin #1841) to avoid conflicts.