Nodes with high miss rate continue receiving inference requests for the rest of the epoch #975
Problem
Nodes with consistently high miss rates (wrong answers, timeouts) remain in the executor pool for the entire epoch. There is no mid-epoch mechanism to stop routing client inference requests to them.
Observed in testnet: nodes with 25%+ miss rates remain active for hundreds of blocks, causing client-visible failures.
Solution in PR #974
Circuit breaker state machine per node: ACTIVE → EXCLUDED → PROBE → ACTIVE
- Exclusion: miss rate > 25% after ≥4 samples (governance-adjustable via ValidationParams)
- Recovery: after cooldown, node gets one probe slot; success → back to ACTIVE
- All state writes in EndBlock — query handlers read-only (Cosmos-safe)
Reputation-adjusted selection: stake weight scaled by reputation score, so well-performing nodes are preferred before hitting the exclusion threshold.
Known issue (addressed in follow-up)
Same-block probe re-exclusion: when a probe succeeds, UpdateCBStateForBlock Pass 2 in the same block could immediately re-exclude the node due to stale miss-rate stats. Fixed in a separate commit — will be included in the new PR.
💬 Comments (1)
🔄 Auto-synced from Issue #975 every hour.
Description doesn't match what's actually in the code and mixes a few things together. Feel free to reopen if you can show a specific case where this happens