DevShard mempool: decoded-message limits and late gossip retention #2249
Local tests reproduce the retention behaviour reported in issue #1899, item 3, including a genuine Finish transaction received again after its signed diff was applied
The tests also support the per-message protection in PR #2226: its parent accepts 65,537 valid encoded transaction entries, while the proposed change rejects the same request before the mempool sink
These are separate properties, the PR bounds one decoded message but does not bound the total retained mempool across messages, this report is not a request to block that narrower fix
Status: REPRODUCED, local regression evidence only, no production exhaustion or throughput claim
Checked revisions
Snapshot: 2026-11-08, Linux amd64, Go 1.25.9, local HTTP loopback and in-memory Host fixtures, no deployed host traffic
| Target | Exact revision | Check |
|---|---|---|
| PR candidate | 22f6ee5f | All four selected tests passed three consecutive runs |
| Candidate parent | 4b8deb65 | Same new fixtures, one run, count-limit assertion fails because the RPC returns success, late-arrival checks pass |
Observations
| Test | Observation | What it establishes |
|---|---|---|
TestResearchMempoolRPCBatchVersusLifetime |
Three accepted batches of 32,768 distinct entries leave 98,304 entries, replaying the last batch adds none, PruneBelow leaves the count unchanged |
Per-message acceptance and cumulative retention are independent, identical hashes are deduplicated |
| Same test, over-count control | A single batch of 65,537 valid repeated entries returns InvalidArgument on the candidate and succeeds on the parent |
The proposed count gate works, rejection is not caused by malformed transaction bytes |
TestResearchLateGossipAfterSignedDiff |
Real Host executes an inference, signed diffs include its ConfirmStart and Finish, mempool becomes empty, a late copy of that Finish remains after signed empty diffs advance to nonce 20 | Normal inclusion removes the entry, late delivery reintroduces it and later diffs do not remove it |
TestResearchMempoolLateArrivalPrimitive |
Arrival before removal leaves zero entries, arrival after removal leaves one | Minimal ordering control for the retention mechanism |
Existing TestSessionHandler_RefusesOverCountDiffsBeforeDecode |
Passes on the candidate in all three runs | Existing diff-count and summed-transaction rejection checks remain green |
The late-arrival Host returns the retained Finish in its response mempool and continues signing, the fixture does not enable the test-only WithGrace checker
The RPC fixture uses the repository's actual local HTTP server, Attach/session-token flow, signed envelopes, decoder, gossip handler, real Gossip and real Mempool, its SessionCore membership decision is a stub returning true, it is not an independent test of membership enforcement or the full production Server adapter
The Host fixture uses real state-machine application and signed diffs with a stub inference engine, delayed delivery enters through OnTxsReceived, it is not a combined end-to-end RPC-to-Host lifecycle test
Reproduction
Use a disposable checkout at the candidate revision, preserve the normal Go and CGO prerequisites for the repository, including its wasmvm library
Companion files to attach alongside this report:
26-003/host_retention_research_test.go→devshard/host/mempool_retention_research_test.go26-003/mempool_retention_research_test.go→devshard/transport/rpcserver/mempool_retention_research_test.go
cd devshard
GOTOOLCHAIN=go1.25.9 go test ./host ./transport/rpcserver \
-run 'TestResearch|TestSessionHandler_RefusesOverCountDiffsBeforeDecode' \
-count=3 -v
Expected candidate excerpts:
signed inclusion retained=0; late identical finish retained=1 after nonce 20, returned in response
over-count response: invalid_argument: ... decode limit ...
three admitted batches: retained=98304; replay unchanged; prune unchanged; over-count rejected
PASS
On the parent revision, run the same new files with -run TestResearch -count=1, the RPC test fails at the over-count rejection assertion and logs over-count response: <nil>, that expected failure distinguishes the candidate's protection from its parent
These are characterization tests, a PASS on retention assertions means the issue was reproduced, not that retention is correct, a future retention fix should invert those assertions around its documented eviction or stale-entry contract
Relevance to existing work
Issue #1899: this adds an executed late-arrival regression to item 3, it does not validate every performance claim in that issue
PR #2226: the candidate's per-request gate is verified against its parent, cumulative retention remains a separate follow-up, a message limit alone is not a lifetime storage limit
PR #1312: signature filtering addresses which transactions enter the pool, the late-arrival example is an authentic Finish already accepted by a signed diff, so signature validation alone would not identify it as stale, this is a design observation rather than a test of that old PR branch
A retention fix needs to distinguish local unsequenced work needed for recovery from imported entries that are already applied or otherwise stale, simply clearing the pool risks dropping valid recovery data, an appropriate regression set should preserve pending legitimate transactions while reclaiming the demonstrated late duplicate
Limits
No OOM, production load, malicious remote access, memory-per-entry estimate or sustained rate was measured, no claim is made that normal signature or settlement processing stops
The earlier exploratory memory estimate counted allocations during insertion and excluded transaction construction, it is not a full retained-memory measurement and is deliberately omitted
The historical signature-filtering PR was read but not built, neither a complete review of it nor approval of the count-limit PR is implied
No new public issue or PR has been created, recheck current PR heads before attaching these results, attach both companion tests so the report remains reproducible
🔄 Auto-synced from Issue #2249 every hour.