Incomplete HTTP 200 streams are cached and replayed as successful responses #1722
Problem
The devshard gateway can store an incomplete streamed chat response in the one-hour response cache. Identical requests then receive the same partial body immediately instead of starting a new inference.
Observed behavior
A production DeepSeek streaming request returned partial reasoning for 1,205.663 seconds and then closed without finish_reason or usage. Eight identical retries returned the same 1,369,407-byte body in 0.871–2.758 seconds. The response fingerprint was identical through two broker endpoints.
A second response included data: [DONE] but had no finish_reason, usage, or final answer. The DONE marker made the response appear complete to clients even though generation did not finish.
No request prompts or credentials are included in this report.
Cause
gatewayChatCacheCapture.cacheEntry calls cacheableResponse, which accepts any non-empty HTTP 2xx body unless it contains a recognized error. It does not validate semantic SSE completion.
This combines badly with streaming paths that can append [DONE] after RunInference returns without a terminal model chunk. A partial response can therefore become a deterministic cached result for one hour.
Expected behavior
A successful streaming response must be cached only when:
- every observed choice has a non-empty
finish_reason; [DONE]occurs after those terminal choice events;- the body has no OpenAI-style error event.
Previously cached incomplete stream entries should also be rejected on lookup.
Reproduction
Pass this HTTP 200 body to gatewayChatCacheCapture.cacheEntry with stream=true:
data: {"choices":[{"index":0,"delta":{"reasoning":"still working"},"finish_reason":null}]}
data: [DONE]
Current result: the entry is cacheable.
Expected result: the entry is rejected because [DONE] is framing, not semantic completion.
Proposed fix
Validate semantic stream completion in the shared cache eligibility path used by capture, Set, and Get. A focused PR with regression tests will follow.
As a separate operational escape hatch, the gateway could honor request Cache-Control: no-cache, no-store. This header must not replace completion validation.
🔄 Auto-synced from Issue #1722 every hour.