Skip to content

`devshard`: Tool-calling / protocol errors → INVALID #1810

Open @tcharchian opened 2026-09-18 22:00 UTC 5 comments Updated 2026-09-29 17:04 UTC

(empty)

💬 Comments (5)

@zpoken commented 2026-09-25 10:41 UTC

Hi @a-kuprin

We are interested in resolving this issue, but there’s currently no description for it. Here’s what we’ve found in the code so far, and we are wondering whether the issue was created around this:

  • Responses containing tool calls are not structurally validated: there is no validation of the JSON in arguments, no check that the tool name exists in the request’s tools, and no consistency check for finish_reason: "tool_calls".
  • Validation replays the generated tokens and compares the logprobs. As a result, malformed tool calls that were actually generated by the model still pass as VALID.
  • A penalty is applied only when vLLM itself returns an SSE error, and in that case it is treated as a miss, not invalid.

Is there already a broader problem statement for this that we could use to write a proposal, or perhaps an existing draft/design vision?

We also have a few questions:

  1. What exactly is meant by “protocol errors”? Is this limited to tool calls, or does it include broader cases such as malformed SSE, invalid JSON chunks, an incorrect finish_reason, or raw tool tags appearing in content?
  2. Should these cases be classified as INVALID or as a miss? This seems to conflict with the principle described in error-finish-miss.md (error = miss). Do we want to change that principle, or introduce a separate class for protocol violations?
  3. Who should be considered at fault when the model itself generates invalid JSON in arguments: the host, or no one?
  4. How do we distinguish client-side responsibility (for example, a bad schema or a tool_choice that the model cannot satisfy) from host-side responsibility, so that a single bad prompt cannot penalize the entire group?
@a-kuprin commented 2026-09-25 18:31 UTC

It is mostly @qdanik theme

@zpoken could you please hold on it, because I just can't deeply explain what is the failure and how we should detect it. Also it is possible that there is hosts with vllm V2, that can give different output. I think looking into the problem as incompatible vllm outputs can give some inside.

For my opinion we anyway need validator to identify is it valid or invalid response. But the general problem that we first need to understand what version generates the tool callings that leads to protocol errors, so it first the log analysis before any development action

@zpoken commented 2026-09-28 10:53 UTC

@a-kuprin Thanks for the information. We’ll investigate this issue further, try different vLLM versions, and analyze the logs to get closer to the root cause.

@zpoken commented 2026-09-29 15:52 UTC

Hi @qdanik,

We reproduced devshard's execute → replay → verdict path on the gonka vLLM builds used on mainnet: 0.25.1 (MLNode 3.0.16), 0.28 (3.1.0) and 0.20. Requests were shaped like devshard's, and verdicts came from the real ExecuteValidation at on-chain thresholds. The model was Qwen3-4B; the mainnet parsers were also tested on CPU. We found several ways honest hosts get INVALID regardless of output quality:

  1. GetEnforcedTokens drops tokens from multi-token SSE chunks. completionresponse.go:137 keeps only Content[0] per chunk, while ExtractLogits keeps all entries.
  2. With speculative decoding on the executor all inferences were INVALID on both 0.25.1 and 0.28, with 30–47% of tokens dropped.
  3. Without it, this occasionally happens on 0.28 (1/20 tool-call responses).
  4. Any host running MTP or speculative decoding would fail every validation.
  5. The min_tokens=64 floor (#1391) produces the malformed outputs, on every version:
  6. invented extra tool calls with tool_choice: "auto";
  7. response_format / structured_outputs / named tool_choice output broken in 15/15 responses. Free text is appended after the JSON, or !!!! padding on 0.20.
  8. Without the floor, everything is clean and replays at similarity 1.0.
  9. These come from honest hosts, so they shouldn't become INVALID as-is.
  10. Version mismatch: 0.25.1 ↔ 0.28 agree in every mode we tried. Only 0.20 disagrees, on a named tool_choice, because it applies the grammar differently.
  11. DeepSeek-V4 prompts render differently on 0.25.1 and 0.28.
  12. At the gateway's default reasoning_effort=max, the rendered prompt is 342 tokens on 0.25.1 and 355 on 0.28, because the "Reasoning Effort" preamble changed.
  13. The validator allows only 3 tokens over the executor's reported InputTokens. So a 0.28 executor checked by a 0.25.1 validator should get "inflated token counts".

The remaining protocol errors (truncated calls reported as tool_calls, markup inside arguments, in content with minimax_m2_append_think) come from the model and parsers. They're identical on 0.25.1 and 0.28 and pass validation.

Numbers and setup - Images: ghcr.io/gonka-ai/vllm:v0.25.1-poc-v4-cu13-hopper-blackwell, v0.28.0-glm53-poc-v2-cu13-hopper-blackwell, v0.20.0-cu129-post2. - Speculative decoding: ngram, 4 draft tokens, executor → validator: - 0.25.1: 50/50 INVALID against a validator without speculative decoding, and 50/50 with it. - 0.28: 50/50 INVALID without speculative decoding on the validator; with it, the replay returns HTTP 500. - DeepSeek-V4 prompt tokens, 0.25.1 → 0.28: | reasoning_effort | 0.25.1 | 0.28 | |------------------|--------|------| | max | 342 | 355 | | high / unset | 264 | 342 | | none | 264 | 264 | - Floor with auto: tool calls went from 24 to 30 (0.25.1) and 24 to 35 (0.20) over 20 prompts. - Parsers: minimax_m2 + minimax_m2_append_think, deepseek_v4, glm47 + glm45. Their behaviour is identical on 0.25.1 and 0.28. - Not factors: VLLM_ENFORCE_STRICT_TOOL_CALLING, strict: true tools, V1 vs V2 runner (dense model only), batched load.
@qdanik commented 2026-09-29 17:04 UTC

@zpoken thank you for the deep research!

  1. It has been fixed and lost during the development. https://github.com/gonka-ai/gonka/pull/1870 - this PR is going to be merged in devshard v6 or even v5 with the host fix release. https://github.com/gonka-ai/gonka/pull/1270 - these changes has been lost during the devshard refactoring
  2. it mostly covers cases when inference less then 64 tokens can be valid with the fraud data. I believe it is a temporary solution until the most effective will be delivered in the next releases

🔄 Auto-synced from Issue #1810 every hour.