Independent review of PoC-decode results #1200
Independently review and re-check the PoC-decode approach from https://github.com/gonka-ai/gonka/issues/1135 and the Axel-T experiments.
Current PoC correlates reasonably well with inference compute, but memory-usage coverage can be improved. The concern is that specialized versions could significantly accelerate PoC without equivalently improving real inference.
PoC-decode proposes extending Proof-of-Compute from prefill to decode steps.
Context
Relevant materials:
- Main issue: https://github.com/gonka-ai/gonka/issues/1135
- Presentation: https://docs.google.com/presentation/d/11zXgKd8q3t7SZ_wqfiMvCqTWqiKxJw65AFmAWDnMcL8/edit
- Experiment artifacts: https://drive.google.com/drive/folders/1tVh6mTsazMfjtSz-J0MTN8KYD5B9g1Bq
- Implementation branch: https://github.com/axeltec-software/vllm/tree/axeltec/poc-decode-proposal
Please use these materials as the source of truth.
Task
Review the PoC-decode proposal and independently re-check the results.
The task should be treated as a critical review, not just a confirmation that the implementation runs.
Review scope
Please check:
- Whether the results from #1135 are reproducible.
- Whether the hypothesis is confirmed beyond the initial experiments.
- Whether the method should be tested on different models.
- Whether the implementation can be integrated into the current vLLM path if the results are confirmed.
- What makes migration painful and how to reduce that pain.
- Whether a safer rollout can start only with new models.
Review points
- Re-check the reported results
Review the experiments from #1135 and the linked Axel-T materials.
Confirm whether the reported PoC-decode results hold under independent review.
- Validate on different models
The current experiments provide the first confirmation of the hypothesis, but independent confirmation on different models is required before integration decisions.
Please identify and run, or define, the minimum additional model checks needed.
- Review the implementation branch
Review the Axel-T vLLM branch:
https://github.com/axeltec-software/vllm/tree/axeltec/poc-decode-proposal
Check whether the implementation matches the method described in #1135 and whether it can be prepared for integration into the current vLLM path if the hypothesis is confirmed.
- Review migration impact
Migration may be painful.
Please identify:
- what makes migration difficult;
- what can be done to reduce migration pain;
- whether rollout can start only with new models;
-
what should be avoided during initial rollout.
-
Critical risk review
Risk level is medium: there is an initial positive signal, but the method still needs an honest critical review.
Please document:
- what is confirmed;
- what is not confirmed yet;
- what requires more experiments;
- what could block integration.
Notes
This task is about independent verification and critical review. If the results are confirmed on different models, the next step can be integration into the current vLLM path.
🔄 Auto-synced from Issue #1200 every hour.