Skip to content

devshard: MLNode capacity is released before streaming response consumption #2180

Open @ilyar opened 2026-10-30 07:01 UTC 0 comments Updated 2026-10-30 07:01 UTC

The standalone inference engine releases the NodeManager slot before the caller consumes the MLNode response body, so a streaming response can still be generating after its slot becomes available again

Source evidence

Checked against the v6 candidate source

  • executeMLRequest passes httpClient.Do as the callback to doWithLockedNode
  • doWithLockedNode calls releaseNode after that callback returns, then returns the response to its caller
  • Streaming response processing happens afterwards in execute.go
  • The passive fallback path also releases its local capacity reservation before returning the response

For a streaming response, receiving HTTP headers is not the end of generation, so this lifecycle does not bound active streaming work by max_concurrent

Expected behaviour

Keep the execution reservation until the response body reaches its terminal state or is closed, release exactly once on success, cancellation and failure, and define how upstream work that continues after client cancellation is accounted for

Reproduction to add

Use an ML endpoint that flushes response headers and one SSE chunk, then waits before completing the body, configure one available NodeManager slot and start a second inference while the first body is still open

Check the ordering of acquire, response headers, release and body completion, the second request must not acquire the same capacity before the first execution releases it

Repeat for the passive fallback path and for response-body read failures

Verification boundary

This report is based on source inspection, the isolated reproduction above has not been executed and the contribution to any particular deployment's overload has not been quantified

The finding is separate from gateway quarantine recovery and speed-up hedging amplification

🔄 Auto-synced from Issue #2180 every hour.