Skip to content

GLM 5.3 Flash: vLLM 0.28 image + PoC fix #1691

Closed @tcharchian opened 2026-08-31 20:23 UTC 6 comments Updated 2026-09-18 18:46 UTC

Pitstop’s main track is MLNode images, experiments, and evaluations for bringing new models onto the network.

The current piece of work is GLM 5.3 Flash: a vLLM 0.28 image plus a PoC fix. PoC currently exceeds the allowed memory budget; that needs to be resolved before handover.

This issue also sets a public bar for how MLNode image work is specified and accepted, so anyone taking an image can see what was checked and what the limitations are.

Confirm the live on-chain set from poc_params.models if the list below has drifted.

Process

  • Experiments completed and written up (hardware, models, what passed, what did not)
  • MLNode starts and reaches a working state (does not crash)
  • Write-up reviewed (review does not have to block image built; once the two items above are done, the image can be built and backward compatibility can be checked in parallel)
  • Image rechecked against the model list below before handover
  • If the image is intentionally for a specific model: notes published before handover, listing which models are supported and which are not

Per-model checks

For every model below: both PoC and inference must pass. Installing this image must not break models already on chain unless that is stated in the notes before handover.

MiniMaxAI/MiniMax-M2.7 — base model, active

  • PoC starts, finishes, and stays within the allowed memory budget
  • Inference works
  • Supported on this image, or unsupported and called out in notes before handover

moonshotai/Kimi-K2.6 — active

  • PoC starts, finishes, and stays within the allowed memory budget
  • Inference works
  • Supported on this image, or unsupported and called out in notes before handover

deepseek-ai/DeepSeek-V4-Flash-0731 — active

  • PoC starts, finishes, and stays within the allowed memory budget
  • Inference works
  • Supported on this image, or unsupported and called out in notes before handover

zai-org/GLM-5.2-FP8 — registered on chain, not bootstrapped (penalty_start_epoch = 500)

  • PoC starts, finishes, and stays within the allowed memory budget
  • Inference works
  • Supported on this image, or unsupported and called out in notes before handover

GLM 5.3 Flash — target of this image (vLLM 0.28)

  • vLLM 0.28 image builds and runs
  • PoC starts, finishes, and no longer exceeds the allowed memory budget
  • Inference works
  • Experiment notes attached (hardware, memory, PoC, inference)

Handover

  • Image tag published
  • Checklist above filled for each model
  • Limitations, if any, written in notes before handover

The image is treated as delivered when this checklist is complete.

💬 Comments (6)

@baychak commented 2026-08-31 20:26 UTC

TY!

@clanster commented 2026-08-31 20:27 UTC

ty

@baychak commented 2026-09-04 14:36 UTC

Status: in progress

Since last

  • Image built and exercised: mlnode-{h100,b300}-glm-5-3-flash:0.2.14-vllm0.28-glm53-test-k3 — vLLM 0.28 + FlashInfer 0.6.18, gonka_poc 0.1.4, no source patches.
  • Measured on 2×B300 TP=2, 4×B200 TP=4, 4×H200 TP=4, 8×H100 TP=8. Reports and raw nonces: kaitakuai/experiments/2026-09/.
  • Memory budget: usable PoC batch ceiling is 32 on every arm — but for a different reason each: OOM on 8×H100, token budget on B300, indexer on B200.
  • 8×H100 does not start at the default --gpu-memory-utilization; it needs 0.95.

Next: sequencing agreed with @clanster on today's sync — update here by 2026-09-05.

Scope

This image targets GLM 5.3 Flash. On 2026-08-31 @vbgd0 and I agreed to prepare it for GLM only and defer the other models, since re-verifying them means re-running everything on rented hardware. The four models above are therefore unchanged and unverified on this image. We propose checking them once the upgrade lands on a release branch — say if you want it the other way.

@baychak commented 2026-09-07 18:51 UTC

Status: in progress; the release line is open.

Since last

Questions, @tcharchian

  • "allowed memory budget" has no definition in the code. Which number do we check against?
  • Checklist ticks in this issue: yours, or may we tick?

Next: after gonka-ai/vllm#106 and gonka-ai/gonka-vllm-plugins#9 merge, @vbgd0 publishes the official image; we rebuild the MLNode image on it and rerun the checklist. Update on merge, and by 2026-09-14 regardless.

@baychak commented 2026-09-11 21:44 UTC

Status: done on our side.

What closes it

  • Proposal 101 passed on 2026-09-10, unanimously: GLM-5.3-Flash added, Kimi-K2.6 and GLM-5.2-FP8 removed from poc_params.models. GLM 5.3 bootstraps at epoch 394 (2026-09-15).
  • The image is @vbgd0's: #1734 carries the release inputs, and he checked the builds on B200 and H200 across the three remaining models.
  • Experiments completed and written up: nine configurations in kaitakuai/experiments/2026-09, including memory behaviour and the working batch ceiling per configuration.
  • Two checklist rows fell away with their models: Kimi-K2.6 and GLM-5.2-FP8 are no longer on chain.

Next: @tcharchian — the work is finished on our side; please close, or say what is missing. We are not ticking the boxes because the question of whether we may tick is open since 2026-09-07.

@baychak commented 2026-09-17 14:26 UTC

Status: delivered; the model is live.

Since last

  • GLM-5.3-Flash entered the epoch on 2026-09-16 at about 02:35 UTC, on the fourth attempt; the 09-15 window did not form — fewer than three hosts.
  • Two incidents on 09-16, both closed in mlnode:3.1.0 (de9150fc): H200 and Blackwell nodes voting each other invalid — first nonce of a batch, gonka-ai/gonka-vllm-plugins#10; a node stuck after /init/generate with no nonces — a replay max_tokens pin inherited from kaitakuai/vllm#21, removed in gonka-ai/vllm#111.
  • H100 runs GLM at PoC batch 8 only, H200 at 16. Our overlay images are rebuilt on 3.1.0 with plugin v0.1.6: kaitakuai/mlnode-{h100,h200,b200,b300}-glm-5-3-flash:3.1.0-overlay-k2.
  • Open since 09-07, no answer yet: the memory-budget threshold and whether we may tick the checklist.

Next: @tcharchian — please close, or say what is missing. Next update 2026-09-21 if the issue stays open.

🔄 Auto-synced from Issue #1691 every hour.