Skip to content

Bug: GET /api/v1/epochs/{N}/participants returns 500 for past epochs (CreatedAtBlockHeight=0) #983

Open @mingles-agent opened 2026-03-31 08:51 UTC 3 comments Updated 2026-08-06 22:28 UTC

Bug

GET /api/v1/epochs/{N}/participants returns 500 Internal Server Error for past epochs. Current epoch works fine.

Repro

GET http://node1.gonka.ai:8000/api/v1/epochs/215/participants
→ 500 Internal Server Error: height must be greater than 0, but got 0

Epoch 215 consistently reproduces this. Any past epoch where CreatedAtBlockHeight was not yet populated will fail.

Root Cause

In queryActiveParticipants (get_participants_handler.go):

  1. First query (no height) fetches activeParticipants
  2. blockHeight := activeParticipants.CreatedAtBlockHeight — for old epochs this is 0 (field was not populated at storage time)
  3. Second call QueryByKeyWithOptions(..., height=0, prove=true) — CometBFT rejects height=0 with the above error

Fix

Check if blockHeight == 0 before the second query. If so, skip the proof query and return the first result directly, with a Warn log for observability.

Fix is implemented in PR #973.

💬 Comments (3)

@bonujel commented 2026-08-03 09:07 UTC

This is still reproducible on current main (4fa6be0, Upgrade v0.2.15 #1497) — but the root cause in the description is not the one that fires, and #973 patches a call site that no longer exists.

TL;DR - The 500 is real and live. - It does not come from the proof-bearing ABCI query. height=0 + prove=true is accepted. - It comes from GetValidatorSetByHeight(Height: CreatedAtBlockHeight), which is fatal. - The handler moved from decentralized-api/internal/server/public/get_participants_handler.go to common/queryapi/epoch.go, so #973 no longer applies to anything.

Reproduction

Drop into common/queryapi/tests/. The stub mirrors the real backend's height contract; each behaviour is sourced in the comment.

package queryapitest

// Repro for #983 against current main (common/queryapi/epoch.go).
//
// Stub behaviour, traced through the pinned deps:
//
//   ABCIQuery(Height=0, Prove=true) -> OK.
//     gonka-ai/cosmos-sdk@v0.53.3-ps17-observability baseapp/abci.go:1256
//     CreateQueryContextWithCheckHeader: `isLatest := height == 0`, and the
//     proof guard is only `height == 1 && prove`. height 0 means "latest".
//
//   GetValidatorSetByHeight(Height=0) -> error.
//     cmtservice/service.go:128 has no `Height < 1` guard (only `> blockHeight`),
//     and forwards to ValidatorsOutput -> getValidators (validator.go:16) ->
//     CometBFT node.Validators(ctx, &0) -> rpc/core/env.go:177 getHeight, which
//     rejects height <= 0 with "height must be greater than 0, but got 0".
//
//   GetBlockByHeight(1) -> OK. The handler asks for CreatedAtBlockHeight+1,
//     which is 1 when the field is 0, and 1 is a valid height.

import (
    "context"
    "fmt"
    "net/http"
    "testing"

    "github.com/cosmos/cosmos-sdk/client/grpc/cmtservice"
    "github.com/golang/protobuf/proto"
    inferencetypes "github.com/productscience/inference/x/inference/types"
    "github.com/stretchr/testify/require"
)

type zeroHeightComet struct {
    cmtservice.UnimplementedServiceServer
    value            []byte
    validatorHeights []int64
}

func (s *zeroHeightComet) ABCIQuery(_ context.Context, req *cmtservice.ABCIQueryRequest) (*cmtservice.ABCIQueryResponse, error) {
    if s.value == nil {
        // Old epoch: the field was never populated at storage time.
        ap := inferencetypes.ActiveParticipants{CreatedAtBlockHeight: 0, EpochGroupId: 1}
        var err error
        if s.value, err = proto.Marshal(&ap); err != nil {
            return nil, err
        }
    }
    if req.Prove {
        return &cmtservice.ABCIQueryResponse{
            Code:  0,
            Value: s.value,
            ProofOps: &cmtservice.ProofOps{
                Ops: []cmtservice.ProofOp{{Type: "iavl:v", Key: []byte("key"), Data: []byte("value")}},
            },
        }, nil
    }
    return &cmtservice.ABCIQueryResponse{Code: 0, Value: s.value}, nil
}

func (s *zeroHeightComet) GetBlockByHeight(_ context.Context, req *cmtservice.GetBlockByHeightRequest) (*cmtservice.GetBlockByHeightResponse, error) {
    if req.Height < 1 {
        return nil, fmt.Errorf("height must be greater than 0, but got %d", req.Height)
    }
    return &cmtservice.GetBlockByHeightResponse{
        SdkBlock: &cmtservice.Block{
            Header: cmtservice.Header{Height: req.Height, ChainID: "gonka-test", AppHash: []byte("apphash")},
        },
    }, nil
}

func (s *zeroHeightComet) GetValidatorSetByHeight(_ context.Context, req *cmtservice.GetValidatorSetByHeightRequest) (*cmtservice.GetValidatorSetByHeightResponse, error) {
    s.validatorHeights = append(s.validatorHeights, req.Height)
    if req.Height < 1 {
        return nil, fmt.Errorf("height must be greater than 0, but got %d", req.Height)
    }
    return &cmtservice.GetValidatorSetByHeightResponse{Validators: nil}, nil
}

// A past epoch whose ActiveParticipants predates CreatedAtBlockHeight being
// populated must not blow up the endpoint.
func TestIssue983_PastEpochWithZeroCreatedAtBlockHeight(t *testing.T) {
    srv := &zeroHeightComet{}
    h := handlersWithInferenceAndComet(t, &stubEpochParticipantsInference{}, srv)

    ctx, rec := echoContext(t, http.MethodGet, "/v1/epochs/215/participants")
    err := h.GetEpochParticipants(ctx, "215")

    t.Logf("GetValidatorSetByHeight called with heights: %v", srv.validatorHeights)
    t.Logf("handler err: %v", err)
    t.Logf("recorded status: %d", rec.Code)

    require.NoError(t, err, "endpoint fails for a past epoch with CreatedAtBlockHeight=0")
}

Result (go test ./queryapi/tests/ -run TestIssue983 -v, Go 1.25.9):

ERROR Failed to get validators subsystem=Participants
      error="rpc error: code = Unknown desc = height must be greater than 0, but got 0"

GetValidatorSetByHeight called with heights: [0]
handler err: code=500, message=height must be greater than 0, but got 0
--- FAIL: TestIssue983_PastEpochWithZeroCreatedAtBlockHeight

Same error string as the report, reached from a different call than the description claims. Note the proof query in the same run returned normally — height=0 + prove=true is not what breaks.

Where it actually breaks

common/queryapi/epoch.go:

  • L142–152 — proof query at Height: CreatedAtBlockHeight. With 0 the SDK resolves it to latest, so no error. (Worth noting separately: the proof is then anchored to the latest app hash while verification at L174–186 compares against block CreatedAtBlockHeight+1 = 1. Verification fails, but it is log-only, so the endpoint would still answer 200 with a proof that verifies against nothing.)
  • L163–170 — GetBlockByHeight(CreatedAtBlockHeight + 1) = height 1, valid, and non-fatal anyway. Fine.
  • L192–198 — GetValidatorSetByHeight(Height: CreatedAtBlockHeight) = 0 → error → return nil, err → 500. This is the one.

The old handler had two fatal zero-height calls (Block and Validators). The move to common/queryapi fixed the first and kept the second.

Why no test catches it

common/queryapi/tests/epoch_participants_golden_test.go hardcodes CreatedAtBlockHeight: 100 (L113), and its GetValidatorSetByHeight stub discards the request entirely (L148, _ *cmtservice.GetValidatorSetByHeightRequest), so the height never reaches an assertion.

On #973

It guards the ABCI proof query, which is not the failing call, so it would not have removed the 500 even when it was written. Separately, the function it patches no longer exists on main — get_participants_handler.go is down to 84 lines and holds only getParticipantByAddress / getAccountByAddress. It needs redoing against common/queryapi/epoch.go rather than rebasing.

Suggested fix

Guard CreatedAtBlockHeight == 0 before the validator-set call in common/queryapi/epoch.go. The design question worth settling first: this endpoint returns ActiveParticipantWithProof, so degrading to a 200 without proof_ops is fail-open on a verification endpoint — a client that does not nil-check would treat unverified data as verified. A 400/404 for epochs that predate the field may be the safer contract. Happy to open a PR either way once the direction is agreed.

@bonujel commented 2026-08-04 00:55 UTC

Correction to my comment above: I wrote that the 500 is "live". That was not supported — the repro forces CreatedAtBlockHeight = 0 through a stub, which shows the code path fails if reached, not that it is reachable.

Checking the public nodes: current epoch is 348 (height 5,377,294), and every past epoch I probed — 1, 5, 50, 100, 200, 215, 300, 330, 340, 344, 346, 347 — returns 404 active participants not found for epoch on both node1 and node2. Only the current epoch returns 200, and its record has created_at_block_height populated (5367703). So on those nodes the zero-height path cannot be exercised at all.

I could not determine why past epochs are unretrievable. Nothing in the module deletes the ActiveParticipants blob (only the ActiveParticipantsSet collection is cleared, and only per-epoch on write), no upgrade handler removes it, and the key format matches what the live proof shows — ActiveParticipants/value/ + big-endian epoch + /, which decodes correctly out of epoch 348's proof_ops. By code reading the blob for epoch 347 should be in state and readable. It isn't. So I can't rule out that an archive node serving historical records would still hit this path.

Net:

  • The root-cause correction stands — the error comes from GetValidatorSetByHeight, not the proof-bearing ABCI query.
  • The note about #973 stands — it patches a call that isn't the failing one, and the function it targets no longer exists on main.
  • Nobody should spend time on a fix until reachability is settled. Withdrawing my offer to open a PR for now.

The larger question this turned up is probably worth more attention than the original report: /v1/epochs/{N}/participants appears to serve only the current epoch, which would make historical participant data — and the proofs over it — unretrievable through this endpoint. If that is intended, this issue can just be closed. If it is not, that is the thing to look at.

@redstartechno commented 2026-08-06 22:28 UTC

Following up on the investigation above: I've opened #1556 fixing the code-level fatal path that was confirmed here — getEpochParticipants no longer sends CreatedAtBlockHeight == 0 to GetValidatorSetByHeight (which CometBFT rejects), and instead degrades to an empty validators array, mirroring the function's existing non-fatal GetBlockByHeight handling.

Deliberately not marked as fixing this issue: the question raised above — why live public nodes return 404 for all past epochs while no delete path for the ActiveParticipants blob exists in the code — remains open and looks operational (pruning/statesync config) rather than code-level. That still deserves its own investigation.

🔄 Auto-synced from Issue #983 every hour.