🔄 Auto-sync: from Discussion #1764 every hour.
Proposal #82: External Test Lab & Community DevNet — M2 Report
Автор: @paranjko · Категория: 
Governance Proposal Reports · Создано: 2026-09-13 12:16 UTC · Обновлено: 2026-09-13 12:16 UTC
📝 Описание
Period: 2026-08-10 — 2026-09-09 · Proposal: #82 (discussion)
Executive summary
Month 2 moved the Lab from setup into active infrastructure, Host lifecycle testing, release qualification, and broker compatibility work.
The rented fleet reached nine GPU hosts, but the 9+ validated MLNodes online target remains Partial: some hosts were deliberately used for destructive JOIN / reset / restore tests, two new NVIDIA hosts were still being prepared at cutoff, and one AMD host remains experimental.
The second QA / infrastructure testing engineer started on September 7 after ~160 responses, 13 interviews, and 2 finalists.
Validation produced concrete results: independent JOIN / recovery testing, immutable DevShard candidate artifacts, the active DevShard v5 / Gonka v0.2.16 qualification campaign, broker compatibility tooling, and a Mainnet gateway v4 regression affecting Kimi-K2.6 non-stream requests (gonka-ai/gonka#1680). The bug was fixed upstream and successfully retested.
On September 9, repeated node5 JOIN testing caused a high-severity Community DevNet consensus halt. Recovery and investigation continued after cutoff in #143.
M2 targets
| Target | Status | Result |
|---|---|---|
| 9+ MLNodes across target regions | Partial | Nine GPU hosts rented; not all qualified / online simultaneously. Some capacity reserved for JOIN / recovery testing. |
| Regional layout | Done | devnet/regions.md |
| Smoke + regression catalogues | Done | smoke · regression |
| Second QA engineer | Done | ~160 responses → 13 interviews → 2 finalists → started Sep 7 |
| Validation work | Done / active | Host lifecycle validation and DevShard v5 / Gonka v0.2.16 qualification active (#28, #49) |
| External Host JOIN | In progress | Public interface available; independent real-server lifecycle validation is not yet stable |
Community DevNet
During M2 the Lab renewed the original fleet and added four GPU hosts, reaching nine GPU hosts plus one network-only host across the US, Finland, the Netherlands, the UK, and a staged host in Russia.
The scale KPI remains Partial because rented capacity is not the same as validated online MLNode capacity. Some hosts were repeatedly rebuilt for JOIN / recovery testing; two new NVIDIA hosts were still in preparation; the AMD RX 9070 XT host remains experimental. See regional layout and live state on gonka-dev.net.
The public JOIN path also advanced from coordinator-driven setup toward an independently usable Host lifecycle: network-derived bootstrap / Join Profile handling, ML qualification under the resolved release, retained diagnostics, backup / restore work, and safer failure reporting. Real-server validation remains active in #28 and #117.
External Testing Team
The second QA hiring round completed with ~160 responses, 13 interviews, 2 finalists, and one selected engineer starting September 7. The new engineer received no payments during M2.
The public M1 strategy is now complemented by smoke and regression catalogues. They expose stable test coverage without publishing campaign-specific credentials, private data, or security-sensitive procedures.
Validation and findings
The main active release-assurance campaign is #49 — DevShard v5 and Gonka v0.2.16 qualification. By cutoff, the Lab had prepared immutable candidate definitions, verifiable artifacts / attestations, separate core and DevShard identities, feature scenarios, and controlled rollout / comparison work. No completed readiness verdict is claimed yet.
Testing of the Coreteam scenarios published in feature has started. Qualification is in progress, no final release-readiness verdict has been issued.
Independent JOIN testing produced reproducible defects and regression inputs. Separately, gonka-ai/gonka#1680 identified a Mainnet gateway v4 regression where Kimi-K2.6 non-stream requests could fail with 502 nonce_finished=false while streaming and v3 remained healthy. The fix was merged into gateway v4 and successfully retested on mainnet-v0.2.15-v4.0.1.
M2 also published broker-side OpenAI compatibility artifacts: Chat Completions host-compat guidance / shim, a POST /v1/responses adapter, and installable skills packaging. See broker-compat/ and skills/.
Incident — GNK-LAB-2026-0001
On September 9, repeated independent JOIN testing for node5 halted block production on gonka-devnet-community; the last block was 306552 at 2026-09-09T18:59:22Z.
Preliminary evidence indicates a consensus-key mismatch during JOIN. With another validator offline, remaining voting power fell below the CometBFT quorum.
Impact: DevNet operations requiring new blocks and JOIN verification halted. No Mainnet impact or loss of funds was detected. Recovery / investigation: #143.
Spending
Invoice amounts are the source of truth; original invoice currency is preserved. Raw invoices remain private because they may contain personal, payment, account, or infrastructure data.
| Budget line | Monthly cap | Spent (M2) | Notes |
|---|---|---|---|
| DevNet machines | $5,000 | $1,964.67 + €299.00 | Renewals + node5–node8; includes staged / experimental capacity |
| Burst GPU rental | $6,500 | $10.00 | SubModel evaluation |
| Tooling and reporting | $250 | $0 | |
| External Testing Engineers | $8,000 | $3,000 | QA work rewards; M1 already reported $2,000 |
| Contingency | $2,250 | $0 | |
| Total | $22,000 | $4,974.67 + €299.00 |
Infrastructure breakdown
| Date | Provider | Configuration | Role | Amount |
|---|---|---|---|---|
| 08-17 | DatabaseMart | RTX Pro 2000 GPU VPS | node4-ml renewal |
$119.00 |
| 08-17 | DatabaseMart | RTX A5000 dedicated | node0 renewal |
$254.77 |
| 08-17 | OneProvider | Tesla T4 dedicated | node1 renewal |
$331.63 |
| 08-17 | OneProvider | network-only dedicated | node4 late-billed prorated service |
$34.33 |
| 08-26 | DatabaseMart | RTX A4000 dedicated | node5, JOIN / recovery target |
$279.00 |
| 08-27 | LeaderGPU | RTX 4090 dedicated | node2 renewal |
€299.00 |
| 08-27 | GIGAGPU | RTX 3090 dedicated | node3 renewal |
$270.03 |
| 09-07 | SubModel | GPU credit | burst evaluation | $10.00 |
| 09-07 | HOSTKEY | RTX 2000 PRO dedicated | node6, staged |
$231.02 |
| 09-07 | GIGAGPU | RX 9070 XT dedicated | node8, experimental |
$229.32 |
| 09-07 | OneProvider | Tesla T4 dedicated | node7, staged |
$215.57 |
Month 3 plan
- Stabilize Host JOIN / backup / restore and publish the incident outcome.
- Complete the 9+ validated MLNodes online target.
- Continue DevShard and protocol updates qualification and publish scoped verdicts.
- Add smoke automation and M3 incident / participant-onboarding artifacts.
- Complete second-QA onboarding and split repeatable validation ownership.