413 on large chat payload due to gateway/host request size mismatch #1658
Summary
Large /v1/chat/completions requests can pass the gateway-side body limit but fail later with 413 Payload Too Large on host transport.
The issue appears to be a mismatch between:
- what the gateway accepts from the client
- what the gateway later sends to hosts
Current behavior
The gateway limits the client request body as raw JSON. The current cap is 10 MiB:
MaxChatRequestBodySize = 10 * 1024 * 1024- client body read/reject path:
readLimitedChatRequestBody
However, after the request is accepted, the normalized request body is stored in:
The host-bound transport request then exposes this prompt as a JSON-facing []byte field:
PayloadJSON.Prompt []byteInferenceRequest.Payload *PayloadJSON- conversion from
host.HostRequestto the transport JSON shape:HostRequestToJSON
In Go JSON serialization, []byte is encoded as base64, which adds roughly 33% overhead.
As a result, a client payload of about 8 MiB can pass the gateway-side limit, but becomes approximately 10.7 MiB when sent to a host as base64-encoded JSON. On top of that, the host-bound request also includes:
diffs- signatures
- envelope / height-sync fields
- other metadata
The host then enforces the actual inbound request body size with the same 10 MiB default cap:
- host transport default cap:
DefaultMaxBodySize = 10 * 1024 * 1024 - host signed POST body is wrapped with
http.MaxBytesReader:VerifyPOSTAuth
This causes the request to fail with 413 on the host side.
Why this is problematic
This is not really a host failure. The request is deterministically too large for the current host transport format.
If the gateway sends the same oversized request to multiple hosts, several hosts may fail in the same way. Redundancy may then treat this as host failure, even though the real issue is that the request cannot fit into the host transport limit after internal serialization overhead.
Related edge case: large catch-up diffs
There is a similar potential issue when a host is far behind and needs a large diff catch-up.
The ordinary inference path sends diffsForHost together with the prompt payload:
diffsForHostreturns all diffs since the host sync nonce- ordinary inference path builds
catchUp := s.diffsForHost(hostIdx):PrepareInferenceFn SendOnlysends that catch-up together withPayload.Prompt:SendOnly
So if a host has accumulated a large backlog of diffs, the host-bound request can exceed the same body cap even when the prompt itself is not huge.
Finalize catch-up already chunks diffs, which may be a useful pattern to reuse:
catchUpChunkSize = 200- chunked finalize catch-up loop:
sendCatchUpWith
Expected behavior
If a request cannot fit into the host-bound wire format, the gateway should reject it before fanout, ideally with 413 Payload Too Large.
This would avoid:
- unnecessary host requests
- false host failure signals
- noisy redundancy behavior
- misleading voting / observability data
🔄 Auto-synced from Issue #1658 every hour.