devshard v7: a context-length refusal at the governance --max-model-len should never cost the host a miss or an invalidation #2078
This is for v7, alongside #1903 (client-caused engine rejections, by @zpoken, moved to v7) and the v7 max-context-length item #1807. The code paths below are devshard/v7 (67230e4dc) and #1903 at b20918017.
Case: a host runs exactly the governance --max-model-len (400000 for DeepSeek-V4-Flash and GLM-5.3, 180000 for MiniMax-M2.7), and a request's prompt plus max_tokens is above it. vLLM rejects it up front with a JSON 400 ("This model's maximum context length is 400000 tokens…"). The host neither lies nor is misconfigured. Other hosts may serve the same request if they raised their limit: a local --max-model-len=600000 survives MergeModelArgs, which compares keys literally, and vLLM keeps the last value. That is #1911, fix open in #1912. Even after #1912 nothing checks the limit a host's vLLM actually runs.
On devshard/v7 today, without #1903: the executor relays the 400 and signs no Finish. For the gateway the message is a retriable capability error (isRetriableCapabilityErrorMessage, redundancy.go:4627). When the request exceeds the table limit, contextRefusalIsFinal (redundancy.go:4356) ends the race, and voteTimeoutsForFailedRequest posts an execution timeout that verifiers accept because no Finish exists. The host ends TimedOut with Missed=1 on every such request. I reproduced this with a gateway race test on d3d806a2c, which has the same code as devshard/v7. #1903 fixes this general case with the 0/0 client-fault Finish.
With #1903 one path remains: validation. replayClientFault (common/validation/validation.go:555) replays the prompt without enforced tokens. A validator with a raised limit serves it and votes Valid:false ("Executor reported a client fault but the prompt is servable"). In the challenge every host replays. With vote_threshold_factor 50, once more than half of the group's slots run a raised limit, VotesInvalid > VoteThreshold and the compliant host's inference is Invalidated (Invalid++), by my reading of resolveChallengeTalliesLocked. On #1903 I tested the steps up to the Valid:false vote, not a full challenge run.
A related hardening point: verifiers accept an error-miss on a client-fault Finish when SiblingServedPrompt (devshard/host/timeout.go:490) holds, i.e. a sibling with the same prompt hash finished with OutputTokens > 0. The rule doesn't check the error type, so it would accept such a miss for a context-length refusal if a gateway sent one. Today's gateway doesn't: it sends an error-miss only for errorTerminal attempts, and a context-length message is not one.
A gateway pre-check doesn't solve this. The gateway sets max_tokens itself (the default request_max_tokens_cap is 4096) but has no tokenizer. It knows only the body size in bytes (gateway.go:2751, "1 token ~= 4 bytes"), so it can't tell a 399k-token prompt from a 401k one. Only a real token count shows whether the request was over the limit.
Proposal: a context-length refusal counts against the host only if the request is shown to fit the governance limit.
- The fix for the validation path: in
replayClientFault, a validator whose replay served doesn't voteValid:falsestraight away. It first counts the prompt tokens itself (/tokenizeon its ML node, or the replay's usage) and confirms the client fault when that count plusrec.MaxTokensexceeds the limit. A forged context-length 400 on a prompt that fits still fails validation, as asked in the earlier review of #1903 and fixed there in d5fbfa38c. - Hardening: in
SiblingServedPrompt, also requiresibling.InputTokens + rec.MaxTokens <= limit(rec.Model). Both values are signed (the sibling's Finish and the request) and checked against the verifier's own applied state, so nothing comes from the refusing host's error text. Actual output is not enough, because vLLM rejects on prompt plus requestedmax_tokensbefore generating: a 397k-token prompt with the defaultmax_tokens4096 is refused at 400k even if the sibling then produced 500 tokens. limit(model)comes from the governance model args, orDevshardEscrow.max_model_lenafter #1763, not from the gateway-onlymodelContextLimits, because verifiers and validators need it too.
Both checks are small on top of #1903. I can open them as a follow-up PR against the branch #1903 lands in, or add them to #1903 if @zpoken prefers.
💬 Comments (6)
@a-kuprin agreed: input + actual output is what has to fit, and validation needs the output logits anyway, so the better fix is that the executor generates instead of refusing.
Today vLLM refuses before generating because it checks prompt + requested max_tokens at tokenization (TokenizeParams, "This model's maximum context length is N tokens. However, you requested M output tokens and your prompt contains K input tokens…"). Later get_max_tokens would already clamp max_tokens to max_model_len - prompt_len, but the request never gets that far. If the ML node checks only the prompt against --max-model-len and lets max_tokens be clamped, a request whose prompt fits returns tokens and logits with finish_reason: length, and validators replay those enforced tokens as usual. No client-fault path is needed for it.
What stays is a prompt that alone is at or above the limit: nothing can be generated, so there are no logits. For that case my #2093 only needs to check prompt_tokens >= limit instead of prompt_tokens + max_tokens > limit. I'll narrow it once the ML node change exists.
I can take the ML node change too (the vLLM fork / plugin side), unless you or @zpoken already have it planned.
Today vLLM refuses before generating because it checks prompt + requested max_tokens at tokenization
I think it's a good way to refuse before generation (we shouldn't spend resources if we are pretty sure it will not fit the context).
So validation part is better to fix: we should have special context length validation path where vllm just tokenizes and checks prompt + requested max_tokens < max_model_len
@a-kuprin #2093 is that path. On a client-fault Finish the validator counts the prompt on its own ML node and confirms the fault only if prompt_tokens + max_tokens exceeds --max-model-len from the epoch's model snapshot. vLLM accepts equality, so the check is >. A prompt that fits keeps Valid:false.
Two differences from "just tokenize":
- The count is the same request with
stream:false, max_tokens:1, so it costs one prefill. vLLM's/tokenizetakes chat messages and returnscountwithout a prefill. The validator can reach it today through the ML node's port 5000 compatibility server (proxy.py, which forwards any path), though that block is marked temporary and the main port forwards only/v1/*. I haven't checked that/tokenizerenders exactly the prompt chat/completions serves for these models; the 1-token request goes through the same path. - The count runs after a replay that served. I can move it in front: over the limit, confirm without a replay; otherwise, or if the count fails or the limit is unknown, replay as now, which covers other client faults and a forged refusal.
I'll make the reorder in #2093 if that matches what you meant, and switch the count to /tokenize if you prefer it over the 1-token request.
I think we can skip prefill phase here and make cheaper validation
@a-kuprin Done in #2093: the count is vLLM /tokenize on the validator's ML node, so there is no prefill and no max_tokens:1 request. Following your note on #1903, #2093 is now standalone on devshard/v7 (the check, the governance-limit resolver and the /tokenize call), so it can land first; #1903 then calls it before the replay.
🔄 Auto-synced from Issue #2078 every hour.
What else I think on this point.
executor that responds with max-context-length error, anyway, should provide logits for generated output tokens. max-context-length >= input_tokens + output_tokens
When we are validating we anyway need logits for output tokens
Overall it looks like fixes need changes to MLNode