[Bug] OpenAI API compatibility: DeepSeek reasoning output concatenated into content & invalid reasoning_effort validation #1654
Summary
When using deepseek-ai/DeepSeek-V4-Flash-0731 via Open Broker (api.openbroker.gonka.gg), the gateway fails to parse reasoning tokens into message.reasoning_content.
Instead, the model's scratchpad/CoT is merged into message.content as plain text without any <think> tags. This breaks downstream agent UI systems (e.g., Nous Hermes, LangChain) because agents ingest their own scratchpad as part of the assistant's final response, polluting dialogue context.
Comparison Logs (Captured on Aug 26)
Prompt:
n2m-probe-1979. What is 23 multiplied by 17? Think step by step, then reply with only the integer.
1. Official DeepSeek API (Expected Behavior)
- Reasoning process is isolated in
reasoning_content. contentcontains strictly the final response.usageproperly reflectsreasoning_tokens.
```json { "choices": [ { "message": { "role": "assistant", "content": "391", "reasoning_content": "We need answer only integer. Need compute 23*17=391..." } } ], "usage": { "completion_tokens_details": { "reasoning_tokens": 34 } } }
- Open Broker Direct Call (Observed Behavior)
⚬ Host/Node: devshard-63078-9577 (vllm-0.25.1)
⚬ CoT and final answer are merged into content.
⚬ No
tags are provided to filter output on the client side. ⚬ reasoning_content is null. { "choices": [ { "message": { "role": "assistant", "content": "23 × 17 = 23 × (10 + 7) = 230 + 161 = 391. \n\n391", "reasoning": null, "reasoning_content": null } } ], "usage": { "completion_tokens": 27, "prompt_tokens": 34, "total_tokens": 61 } }
💬 Comments (5)
Sorry — the original "reply with only the integer" line was only a fixture so the two API shapes are easy to compare. We are not asking to change min_tokens.
Bug: Open Broker does not split DeepSeek-V4-Flash CoT the way the official API does.
- Official: CoT →
message.reasoning_content, answer →message.content - Broker: CoT concatenated into
content;reasoning_content/reasoning=null; no<think>tags
OpenAI-compat agents persist message.content as assistant history, so the scratchpad pollutes the next-turn prompt.
Same prompt, captured 2026-08-29:
What is 14 multiplied by 16? Think step by step, then reply with only the integer.
Official DeepSeek V4 Flash:
"content": "224",
"reasoning_content": "We need answer question. Need think step by step then reply with only integer. ... So final \"224\" only."
Open Broker direct (deepseek-ai/DeepSeek-V4-Flash-0731, vllm-0.25.1, reasoning_effort: "high"):
"content": "14 × 16 = 14 × (10 + 6) = 140 + 84 = 224.\n\n224",
"reasoning_content": null,
"reasoning": null
Top-level reasoning_effort: "high" does not split. chat_template_kwargs: { "thinking": true, "reasoning_effort": "high" } does on some hosts (uneven --reasoning-parser / vLLM V2).
How do you want this solved? Document chat_template_kwargs as the contract? Map top-level reasoning_effort like official DeepSeek in the new gateway? Required host flags / parser version? We are not asking for a broker-side rewrite of responses.
@dmrtest The host can decide how to run the model and some of them does not start the vllm with the --reasoning-parser parameter. That's why you can get only content w/o reasoning field.
min_tokens: 64 - it means if you asked to reply only integer and your reasoning tokens length will be less than 64 tokens - you will see the hallucinations in the content field. JFYI
--reasoning-parser is should be enabled on every MLNode, now it isn't because the host decided to not start model with reasoning parser.
Thanks — that matches what we see: no --reasoning-parser → CoT stays in content, reasoning_content is null.
min_tokens: 64 noted. The concatenated samples are unparsed CoT in content, not padding to 64 tokens. The agent bug is the missing split on parser-less hosts.
If --reasoning-parser is required for this model, when does it become mandatory on every MLNode? Until then the OpenAI-compat contract is a host-lottery and not a drop-in for official DeepSeek V4 Flash.
@dmrtest It is required for this model but in case if the host decide to run own config without reasoning parser - it will work. I guess protocol can validate this gate but no plans for now.
🔄 Auto-synced from Issue #1654 every hour.
reasoning_effortwill be available in the new gateway.min_tokensis currently 64). Dropping that to 1 would make a short numeric answer possible, but it is a protocol tradeoff (validation / accounting)--reasoning-parser/ vLLM V2 support).There is no broker-side workaround that looks safe to ship. The remaining question is a design one — whether to keep the validation floor or treat
min_tokens: 1as an explicit tradeoff. That needs a discussion before anyone can promise the official DeepSeekreasoning_contentsplit. Happy to have community input on that tradeoff; @qdanik please correct if anything is wrong