Remove automatic full-response caching so repeated chat requests start a new inference #2265
Proposal
Permanently remove automatic full-response caching for chat completions based on matching model and normalized request body. A new client POST should start a new inference attempt even when its body matches a previous request.
This concerns the gateway chat response cache, not vLLM prefix/KV caching or replay of the result of an existing protocol operation.
Related issues and pull requests
-
1722 reported incomplete HTTP 200 SSE responses being cached.
-
1723 was superseded by #1827. #1827 has been merged into devshard-0.2.x-v6 and checks response completion before caching.
-
1864 proposed caller-scoped cache entries, but preserved retry deduplication and was closed without merging.
This proposal changes the API contract rather than duplicating the completion-validation fix. A correctly terminated response can still be unusable to the application.
Problem
An agent client may repeat a request because the model failed to call the output tool, returned an unusable answer, or did not follow the task. That retry needs a fresh generation attempt. An identical request body does not establish that the client wants the previous result.
The current cache conflates two different intentions:
- Retrieve the result of an already executed operation.
- Start another attempt with the same inputs.
The Response cache and request accounting section of devshard/docs/proxy-architecture.md explicitly describes serving equivalent requests without consuming another devshard nonce. In the inspected main checkout (8af3f3f692c2809f7c61f4f20cb4fa22d6ec9eee), response_cache.go defines a one-hour TTL and keys entries by model, response-format intent, and normalized request body.
Checking finish_reason, JSON, and SSE structure can reject protocol failures, but cannot generally determine whether an answer is useful or correct for the application. A response with finish_reason: stop may still require another attempt. Caching it makes a retry reproduce the failure instead of allowing recovery.
The project's own tests already account for this: WaitGatewayChatExpectFailure in devshard/testenv/citest/harness/adversarial.go changes the request body on each attempt so an early HTTP 200 cannot freeze subsequent checks on cache hits. Ordinary clients should not need to add random prompt text or change semantic parameters to obtain a new execution.
Motivating observations
Agent requests to DeepSeek-V4-Flash-0731 returned no tool calls despite an outgoing tool_choice: required and a token budget of 16384. Responses reported usage of 512 input / 128 output tokens. Two consecutive attempts for the same agent returned identical unfinished reasoning. The saved response summaries contained finish_reason: stop.
These observations do not prove a cache hit or identify the component responsible for the original bad response. Broker and host logs would be needed to establish that. They illustrate why clients need a fresh attempt even after a formally successful response. Underlying host execution failures should be investigated separately.
Expected behavior
- Permanently remove automatic replay of a full response based on the body of a new chat-completions request, rather than merely shortening the TTL or temporarily disabling the cache.
- Two separate identical client POSTs should start two separate inference attempts. Their generated text need not differ; the requirement is fresh execution rather than cache replay.
- Preserve prefix/KV caching to accelerate prompt processing.
- Preserve replay of the same protocol operation where needed for reconnection and response-hash consistency. That is not a new inference and is outside this proposal.
- If the API needs client-operation deduplication, define an explicit operation-identifier contract separately instead of inferring identity from request contents.
Verification
A regression test should send two identical client POSTs and assert two fresh runtime executions. Re-delivery of a single existing protocol operation must still avoid re-running the model.
The trade-off is additional compute for requests that would previously have been cache hits. For agent recovery, this is the intended cost of another attempt. Normal conversation turns change the history, while repeated prefixes can still benefit from KV caching.
💬 Comments (1)
🔄 Auto-synced from Issue #2265 every hour.
you can use different
seedbetween requests to get a new response