devshardd: a replay that the validator's own ML node aborts at PoC start is voted invalid against the executor #2297
What happens
When the validator's ML node starts PoC generation, every replay still running on it ends early. If the replay comes back as a complete stream (usage chunk and [DONE]) with fewer output tokens than the executor's enforced tokens, devshardd scores it like any other replay: "Inflated token counts." for a short replay, "Executor returned error but prompt is servable." for an empty-sentinel one. Both are a Valid: false vote against an executor that did nothing wrong.
Seen on a validator running devshardd v6.0.0 at a confirmation PoC on 2026-11-18, generation start 12:29:13 UTC. common/validation/validation.go is the same blob (5b74c871) in v6.0.0 and on the current devshard/v7 tip 34ddd777, so v7 behaves the same.
- 384
validation failed: inflated token countswarnings between 12:29:14.024 and 12:29:25.346 (373 distinct inference ids), none in the 9 minutes of log before. 579 in total by 12:35. - In all 384 the input matched (
validationInput == claimedInput); only the output was short. Mean claimed output 10,752 tokens, mean replayed 2,436; 42 replays returned 0 output tokens. - No
validator replay stream failed on ML nodeline in the whole log, so every one of these replays was a complete stream and none was rotated to another node. - Of the 579 warnings, 234 were followed within 10 s by
validation tx published ... result_valid=falsefor the same inference id and 167 by a later invalid vote; 6 were later voted valid and 172 had no vote in the log window. Example: escrow 175729, inference 710, claimed output 402, replayed 0, warning at 12:29:14.024, invalid vote at 12:29:14.026. - Invalid votes per minute: 10 to 41 (about 22 on average) from 12:20 to 12:28, then 135 invalid and 5 valid at 12:29. Not every invalid in that minute comes from a cut replay, but the 234 above follow an inflated-token warning for the same id.
The log does not record the replay's finish_reason, so that these replays ended with "abort" is not observed here. It is what the plugin's abort_all_requests path produces according to #2146 and gonka-ai/gonka-vllm-plugins#34, and the timing matches.
Where
NormalizeReplayStreamfails a replay only on anerrorevent or a missing[DONE]; a stream with an"abort"choice passes as complete: validation.go#L213-L266ExecuteValidationWithPolicyparses the replay and goes straight to the empty-sentinel and token-count checks, with no look atfinish_reason: validation.go#L551-L579InvalidInferenceResultbecomesfalse: validator.go#L393-L396, published asMsgValidation{Valid: result.Valid}: host.go#L1732-L1736- The path that should apply instead already exists:
ErrValidationDeferredkeeps the obligation and retries after the 30 s cooldown: validate.go#L613-L614, host.go#L1677-L1681, host.go#L117
Why it is wrong
finish_reason: "abort" is set by the validator's own vLLM when it drops the request. It says the replay was not finished, not that the executor's tokens are wrong, so it cannot be the basis of a vote. Today the executor takes the invalid for the validator's PoC pause, and invalid votes feed the consecutive-invalid counter and the invalidation-rate test.
Proposed fix
In ExecuteValidationWithPolicy, right after the replay is parsed and before the empty-sentinel and token-count checks, return a new ErrReplayAborted if any choice of the replay has finish_reason == "abort". classifyExecuteValidationErr wraps it in ErrValidationDeferred, so the host retries after the existing cooldown and no vote is cast. The deferral keys only on that node-set reason: a short replay that ends with "stop" or "length" is still voted invalid, as today. Also add the replay's finish reasons and the enforced token count to the inflated token counts warning, so the next PoC shows which reason the short replays carried.
gonka-ai/gonka-vllm-plugins#34 fixes this at the source by turning an aborted request into a 503 error event. This check covers validators whose ML node does not run that plugin version yet, and defers at once instead of rotating through other nodes that may be in PoC too. Not covered: an aborted replay is an HTTP 200, so the validation credit for that attempt is still spent.
Related: #2146 (executor side of the same abort), gonka-ai/gonka-vllm-plugins#34, #1805.
💬 Comments (1)
🔄 Auto-synced from Issue #2297 every hour.
Starting work, PR in a day or two