First full MNode image for Decode PoC (DeepSeek): testing + coefficients #1690
Decode PoC thresholds and expert seeding for DeepSeek. DeepSeek seeding is more complex than MiniMax; the scheme needs a fix before coefficients alone are useful. Kimi Decode PoC thresholds stay on hold (unclear future of Kimi on this path). MiniMax image verification of the current implementation is #1689. Integration / design questions are #1688. Confirm the live on-chain id from poc_params.models if it has drifted. Target: deepseek-ai/DeepSeek-V4-Flash-0731.
Process — Experiments completed and written up (hardware, models, what passed, what did not) — Seeding scheme for DeepSeek is described (how it differs from MiniMax) — Scheme fix is written up before threshold collection is treated as done — If work is DeepSeek-only: notes before handover, stating that Kimi is out of scope
Expert seeding (DeepSeek) — Current seeding scheme is documented — Why MiniMax seeding is not sufficient for DeepSeek is written up — Scheme is fixed or a concrete proposal is on this issue — Weight distribution across seeded experts is measured — Closer-to-uniform weights across seeded experts are shown, with notes if uniform is not possible
Thresholds (DeepSeek) — Decode PoC thresholds for DeepSeek are collected after the seeding scheme is stable — How thresholds were measured is written up (hardware, setup, logs / notebooks) — Thresholds are not treated as acceptance of a MiniMax-style full image — that bar is #1689
Kimi — Kimi Decode PoC thresholds are explicitly out of scope on this issue (on hold)
Handover - [ ] Seeding scheme notes published - [ ] DeepSeek thresholds published, or listed as blocked on the scheme fix
💬 Comments (6)
ty
Dependency: not blocked by #1688. @tcharchian — for the ordering.
- The DeepSeek seeding scheme lives on this issue. #1688 says so:
the scheme fix itself is #1690. - Blocking this issue on it would be circular: #1689 waits on the algorithm freeze, and that freeze includes this scheme.
- The work is not blocked; publishing the numbers is. The scheme change for the hash-routed layers (pseudo token ids for the seeded decode steps, natural gate logits on those layers) and the tau-grid thresholds on vLLM 0.25.1 are measured on our forks.
- Thresholds are posted here after the scheme note is written and the constants are signed off. They are consensus parameters, and we do not publish them before they are on chain.
Ordering inside this issue is unchanged: thresholds count as done only after the seeding scheme is fixed and written up.
Next: scheme note for the hash-routed layers, on this issue by 2026-09-08. Status update every Monday, next 2026-09-07.
Status: in progress. The seeding-scheme note is written; the 2026-09-08 date was missed.
Since last
- The scheme is fixed and described: pseudo token ids are always on for every model, hash-MoE gates keep their natural weights, an explicit model list guards that branch, and a model outside the list carrying an integer table on its gate is refused at attach. The note follows in a separate comment.
- Why the MiniMax scheme was not enough: PoC rows carried token id 0, so the three token-id-routed layers executed the same 6 experts out of 256 for every nonce and every step, leaving the other 250 per layer unexercised. After the fix the ids vary per nonce and per step.
- Uniform weights across those experts are not reachable: the model's own gate supplies the weight, and forcing it drove the weight to zero and made the coverage hollow. The per-layer histogram of attested experts after the fix is not measured yet — one instrumented run on one card.
- Kimi is out of scope rather than on hold: the model is out of the chain, and GLM 5.3 takes its slot.
Next: the note as a comment here; DeepSeek thresholds after the choice of statistic and the sm80 policy are settled. Decode dist_threshold values are already in @vbgd0's #1743. Update here 2026-09-14.
Seeding scheme for DeepSeek-V4-Flash under decode-PoC
1. How the seeding works today
A nonce is 256 prompt tokens and 256 decode steps. Each step stores the index of the nearest of 12 reflection vectors; the chain of those indices is the artifact.
Two things are seeded from the block hash and the nonce, so that a prover cannot choose what the model computes:
- the reflection vectors, drawn per block hash and per nonce;
- the MoE routing: on every routed layer the gate's expert choice is restricted to a seeded, step-rotating window, so a nonce sweeps experts instead of resting on the few the model prefers.
DeepSeek-V4-Flash has 43 layers and 256 experts with top-6 per token. Forty layers are
ordinary routed layers and take the seeded window. Three layers do not: they pick experts
by token id through an integer table, tid2eid[vocab, 6], plus one shared expert. The
gate on those layers supplies the weights, not the choice.
2. Why the MiniMax scheme is not enough here
MiniMax has no token-id-routed layers, so seeding the router covers the model.
On DeepSeek it does not. PoC rows carried token id 0 in every position and on every step,
because a PoC row has no real prompt. Row 0 of tid2eid is a constant, so the three
token-id layers executed the same 6 experts out of 256 for every nonce and every step.
The other 250 experts on each of those layers were never exercised by PoC, and experts are
not shared between layers.
That is room to cheat: a prover can drop or damage the 250 unexercised experts per layer and no nonce will notice, while chat traffic that does use them degrades.
3. The fix
Three parts, all in the plugin:
- Pseudo token ids, always on, for every model. A PoC row's prefill phase gets ids from
derive_pseudo_input_ids(the derivation the prefill scheme already uses); each decode step gets its id from the chain state (base, step, previous index) under its own salt. The id reaches only the token-id table; the row's embedding stays seeded. With the token-id branch disabled the substitution is bit-identical to the old constant 0, so MiniMax reference artifacts are unchanged (verified numerically). - Token-id layers keep natural gate weights. Forcing the seeded window on those gates
set the weight to
sqrt(softplus(-1e4)), which is zero: the expert output was multiplied by zero and never entered the trajectory, so the new coverage would have been hollow. The choice stays deterministic through the table; the weights come from the gate's own logits at the table indices. - An explicit model list guards the natural-weight branch.
_TOKEN_ID_ROUTED_MODELScurrently holdsdeepseek_v4. A model outside the list that carries an integer 2-D table on its gate is refused at attach with a message naming the list, rather than being handled silently. A unit test covers the refusal on a synthetic gate.
Structural auto-detection was considered and rejected: a model shipping a similar table
would silently change how PoC attests it, and that is a consensus property. An explicit
list forces a human to look at the model first. The boot log states what happened:
hash-MoE gates left natural: 3 and 40 MoE routers seeded.
4. What the experts see, before and after
Before the fix, per token-id layer: 6 of 256 experts, the same set for every nonce and every step, and their contribution carried zero weight — nothing was attested at all.
After the fix the ids vary per nonce and per step, so the table picks vary through a run, and the weights are the model's own gate scores at those indices. Two consequences worth stating plainly:
- coverage is now bounded by how many distinct ids a run produces, not by a constant;
- the weight distribution across the seeded experts is not uniform and cannot be made uniform: on these layers the model, not the scheme, decides the weight, and forcing it is exactly what produced the zero-weight hollow coverage.
We have not yet measured the per-layer histogram of attested experts after the fix. What is checked today is the boot-log assertion above and a regression that the verdict is unchanged. Measuring the histogram is one instrumented run on one card; say the word and it goes into the next campaign.
Status: in progress. The 09-14 update was missed.
Since last
- No questions or objections on the seeding-scheme note (09-11).
- The per-layer histogram of attested experts after the fix is still not measured: one instrumented run on one card, and no box is up for our work right now.
- DeepSeek thresholds still wait on two decisions — the choice of statistic and the sm80 policy; neither is taken. In the 0.25.1 report A100 appears as a validator while the policy is not fixed.
- DeepSeek re-measured on 0.28 (09-15/16): +17–20 % PoC over 0.25.1 everywhere except H200. For the decode-PoC release DeepSeek stays on 0.25.1 (agreed with @vbgd0 on 09-16), so thresholds are collected on 0.25.1 — the re-measurement does not invalidate them.
Next: a proposal for the statistic and for sm80 — here, by 2026-09-21; the histogram as soon as a box is up. Update here 2026-09-21.
🔄 Auto-synced from Issue #1690 every hour.
TY!