#8 – Upgrade Proposal: v0.2.4
Passed
Proposal ID: 8
Type: Software Upgrade
Submit: 2025-10-22 05:46 UTC
Voting: 2025-10-22 05:46 UTC → 2025-10-22 08:46 UTC
Upgrade Proposal: v0.2.4
Full Proposal
Full proposal — click to expand · Open in separate page →
Upgrade Proposal: v0.2.4
This document outlines the proposed changes for on-chain software upgrade v0.2.4. The Changes section details the major modifications, and the Upgrade Plan section describes the process for applying these changes.
Upgrade Plan
This PR updates the code for the api and node services and modifies the container versions in deploy/join/docker-compose.yml.
The binary versions will be updated via an on-chain upgrade proposal. For more information on the upgrade process, refer to /docs/upgrades.md.
Existing participants are not required to upgrade their api and node containers. The updated container versions are intended for new participants who join after the on-chain upgrade is complete.
Proposed Process:
1. Active participants review this proposal on GitHub.
2. Once the PR is approved by a majority, a v0.2.4 release will be created from this branch, and an on-chain upgrade proposal for this version will be submitted.
3. If the on-chain proposal is approved, this PR will be merged immediately after the upgrade is executed on-chain.
Creating the release from this branch (instead of main) minimizes the time that the /deploy/join/ directory on the main branch contains container versions that do not match the on-chain binary versions, ensuring a smoother onboarding experience for new participants.
New MLNode container v3.0.10 is fully compartible with v3.0.9 and can be updated asyncronously at any time.
Important: to support new features as models auto-downloading, HF_HUB_OFFLINE should be disabled (env variable removed from deploy/join/docker-compose.mlnode.yml).
Testing
Testnet
The on-chain upgrade from version v0.2.3-patch2 to v0.2.4 has been successfully deployed and verified on the testnet.
We encourage all reviewers to request access to our testnet environment to validate the upgrade. Alternatively, reviewers can test the on-chain upgrade process on their own private testnets.
Changes
Training Security
See the commits related to this change here and here
Overview
Training in Gonka is not ready for broad use. However, all training messages are currently available, causing both confusion and an attack surface for DoS attachs
Solution
All training related messages are now behind two allow lists (EXEC to run training tasks, START to initiate training). The lists are controlled via governance votes.
This also addresses several Certik audit issues.
Major Certik Audit fixes
This is a set of fixes that address issues found in the Certik Audit that required more substantial fixes in the code. References to the specific Certik issue are in parenthesis The overall changes can be seen here
Pubkey must match address
When adding a new Participant, the pubkey must match the address (GOC-12). Specific changes here
Panic avoidance during EndBlock
Panics during EndBlock will cause consensus failure. These are a set of changes to avoid possible panics, either explicitly called or through methods starting with Must. (GOC-13, GOI-24)
Changes are here and here
Validate inference timestamps on chain
While replay attacks are primarily avoided via Signature dedupe for InferenceId, after pruning older inferences could be replayed. This adds on-chain checks for old inferences being replayed. (GOI-01). Changes are here
Pruning Improvements
The first version of pruning was crude and liable to issues as scale increased as all pruning would take place in a single block. This introduces a more scalable version of pruning that will prune a given number of items each block until all items are pruned, and introduces a generic Pruner that can be re-used for this logic for other items as needed. (GOC-14). Changes are here and here
Validation Limits
MsgValidate had zero limits, allowing even non-participants to submit invalidations, requiring and expensive chain-wide revalidation each time. Even inadvertent invalidations could cause significant strain on the chain, and have caused one chain halt.
There are several fixes to address this:
1. No longer include the ResponsePayload in MsgValidate (this was 90%+ of the message size)
2. Check that a validator is an active participant AND has the model for the Inference being validated
3. Add limits to the number of Invalidations allowed for each Participant (based on Power and Reputation)
Config Management Improvements for API nodes
Config Storage
Config Management was entirely file based for API nodes. While this was adequate for infrequently or never changing attributes such as URLs, account addresses or network settings, it is not performant or safe for frequently changing values such as block height, node data or seed info.
Solution
Introduce a file based SqlLite DB to handle changing values and synchronize them for the API node.
Source code is here
MLNode management api
REST API is becoming new main way to manage MLNodes. node-config.json is used only at the first load.
New endpoint PUT /admin/nodes/:id is introduce to update MLNode infor without deleting.
Unordered Transaction Timeout fix
Context
We use unordered transactions. Each transaction has a TTL—the time window within which the transaction must be executed. Some transactions are sent with a retry: we send repeatedly until we confirm that the transaction has made it to the chain
Problem
Under heavy load or other extreme circumstances, the chain blocks can begin to slow or even halt. Since transactions are compared with block time for TTL and signed with node machine time, the drift would result in all messages being rejected as being ahead of block time. They are then resent, further propagating the error and resulting in additional strain on the network as it tries to recover.
Solution
- Use the latest block time instead of node machine time to sign and verify TTL for transactions
- Detect slow or halted chains and stop sending retries to allow better chain recovery
- Cap number of retry attempts (at 100 for now)
Source code is here
Support small GPUs on TestNet
Problem
testnet should support smaller GPUs in order to encourage bigger participation in testnet, increasing tests and therefore network robustness.
Solution
- Add testenv specific environment variables to detect when the chain is testnet vs mainnet
- Add testenv specific params to have different proof-of-compute configurations (enabling smaller GPUs and faster testing) Full changes are here
MLNode Management APIs & vLLM Stability Improvements
Far more details available in the PR description at the top here, but a summary: 1. Add new Endpoints to the ML Node that allow checking status of model downloads and other properties 2. Add new Endpoints to the ML Node that checking GPU status and drivers 3. Improvements in vLLM Stability deployment, as well as additional status APIs to check on progress 4. Integration of the above with the API node, with background pre-downloading of upcoming epochs and updating hardware on the chain.
Health endpoints for MLNode and experimental auto-detection for node setup issues here
Final Tally
Messages
| # | Type |
|---|---|
| 1 | /cosmos.upgrade.v1beta1.MsgSoftwareUpgrade |
Contract Details
[
{
"@type": "/cosmos.upgrade.v1beta1.MsgSoftwareUpgrade",
"authority": "gonka10d07y265gmmuvt4z0w9aw880jnsr700j2h5m33",
"plan": {
"name": "v0.2.4",
"time": "0001-01-01T00:00:00Z",
"height": "937700",
"info": "{\n \"binaries\": {\n \"linux/amd64\": \"https://github.com/gonka-ai/gonka/releases/download/release%2Fv0.2.4/inferenced-amd64.zip?checksum=sha256:ea00bbc6a40aab85ec0192851a800ac803c8f9513fa4bac8b75545aeacd3bf64\"\n },\n \"api_binaries\": {\n \"linux/amd64\": \"https://github.com/gonka-ai/gonka/releases/download/release%2Fv0.2.4/decentralized-api-amd64.zip?checksum=sha256:0a26945bc43bd8be11538197cf78d3689fa7d46d0b6eb7ee997d53079feef2b0\"\n }\n }",
"upgraded_client_state": null
}
}
]