Skip to content

Large per-host catch-up diff backlog can cause 413 on host transport #1660

Open @aikuznetsov opened 2026-08-27 13:24 UTC 0 comments Updated 2026-09-18 21:32 UTC

Summary

A host can fall far behind the current session state and later receive an oversized catch-up request, causing 413 Payload Too Large on host transport.

This is related to large payload handling, but the root cause is different from the raw client prompt size issue. In this case, the main factor is the accumulated per-host diff backlog.

What happens

The session tracks a separate sync cursor for each host:

hostSyncNonce map[int]uint64

This cursor represents the latest nonce that a specific host has successfully acknowledged.

When the gateway prepares a request for a host, it builds catch-up diffs using diffsForHost:

diffsForHost

That function returns all diffs with Nonce > hostSyncNonce[hostIdx].

The cursor only advances after a successful host response is processed:

processResponse updates hostSyncNonce

So if a host is unavailable, times out, fails state verification, or repeatedly receives a transport-level error such as 413, its sync cursor does not move forward while the session continues accumulating new diffs.

Ordinary inference path

When that host is selected again for inference, the ordinary inference path builds the full catch-up backlog:

catchUp := s.diffsForHost(hostIdx)

Then it sends the catch-up diffs together with the current prompt in one host-bound request:

SendOnly sends Diffs: p.catchUp and Payload.Prompt

So the host-bound body size includes the catch-up diffs, prompt payload, height-sync / envelope fields, signatures, metadata, and JSON/base64 overhead.

Why this can produce 413

Each diff is represented in the JSON transport as DiffJSON:

DiffJSON

DiffJSON contains several []byte fields:

  • Txs
  • UserSig
  • PostStateRoot

In Go JSON serialization, these byte slices are encoded as base64, which adds overhead to every diff.

So a large backlog, for example thousands of diffs, can become large enough to exceed the host transport body limit even before considering the prompt.

The host transport currently enforces a 10 MiB default body cap:

DefaultMaxBodySize = 10 * 1024 * 1024

VerifyPOSTAuth wraps the request body with http.MaxBytesReader

Validation and vote diff size

Validation and voting traffic can make the backlog heavier.

For example, these tx types include signatures, escrow IDs, hashes, and other metadata:

MsgFinishInference

MsgValidation

MsgValidationVote

So the issue is not only the number of diffs, but also the average serialized size of each diff.

Failure loop

This can create a self-reinforcing failure loop:

  1. Host falls behind.
  2. Gateway builds a large catch-up for that host.
  3. Catch-up plus prompt exceeds the host transport body limit.
  4. Host returns 413 before applying the diffs.
  5. Gateway does not advance hostSyncNonce.
  6. The next attempt sends the same or even larger backlog.

At that point the host may be unable to catch up through the ordinary inference path.

Existing partial mitigation

There is already chunked catch-up logic:

catchUpChunkSize = 200

sendCatchUpWith sends catch-up diffs in chunks

The comment explicitly mentions that large sessions can accumulate many diffs and that sending them all at once risks timeouts and oversized request bodies.

However, this chunking is not used in the ordinary inference path. Ordinary inference currently sends catchUp + prompt in a single POST.

A similar risk exists in heartbeat / height-sync flow, where diffsForHost is also sent as one request:

heartbeat sendComposedDiff uses s.diffsForHost

This request does not include the prompt, but a sufficiently large diff backlog can still exceed the same transport body limit.

Expected behavior

If a host is too far behind to receive its catch-up in a single host-bound request, the gateway should avoid sending an oversized ordinary inference request.

The system should not treat this as an ordinary host failure, because the request is deterministically too large for the current host transport format.

This would avoid:

  • repeated 413 failures
  • stale hostSyncNonce cursors
  • self-reinforcing catch-up backlog growth
  • misleading host failure / voting signals
  • unnecessary fanout of requests that cannot fit into host transport limits

🔄 Auto-synced from Issue #1660 every hour.