Skip to content

[BUG] Incomplete JSON clock timestamps produce a false divergence sample #1965

Open @EazyHood opened 2026-10-09 21:02 UTC 0 comments Updated 2026-10-09 21:02 UTC

Summary

common/probe treats a JSON clock response with send_unix_ns but no usable recv_unix_ns as a valid four-timestamp sample. In a deterministic local reproduction, it reports a clock offset of -895,773,600 seconds with no error. A usable HTTP Date header does not prevent the false sample.

Motivation and impact

This is an observability correctness issue for operators using the optional JSON clock-response format. An incomplete response can produce a misleading clock-divergence metric instead of falling back to Date or omitting the sample. It is an edge case, not a claim of a production incident or network-wide failure. The normative two-header response is not affected; probe results are observability-only, not routing or consensus inputs.

The clock contract specifies both JSON timestamps, with no optional fields. The parser already rejects incomplete timestamp headers.

Reproduction and cause

Checked on main fd99f88d1f5d51f498a7108a56c41a826aa0dbe9, using Go 1.25.9.

  1. Set the injected probe clock to 2026-10-09T12:00:00Z.
  2. Return HTTP 200 with Content-Type: application/json, no timestamp headers, and body {"send_unix_ns":1791547200000000000}.
  3. Call ProbeOnce with this endpoint as ClockURL.

Observed:

up=true hasDivergence=true source=clock divergenceSeconds=-895773600.000 err=<nil>

The same happens with "recv_unix_ns":null. Adding Date: Fri, 09 Oct 2026 12:00:00 GMT still produces the false clock sample. Supplying both timestamps correctly produces zero offset.

In parse.go, pingJSON has plain int64 fields. An absent/null receive timestamp becomes zero. The both == 0 check does not catch it, and send >= recv passes. The function returns both presence flags as true; finishPing then uses epoch zero as T2 and skips the Date fallback.

Minimal regression test, saved as common/probe/partial_json_regression_test.go:

package probe_test

import (
    "common/probe"
    "context"
    "io"
    "net/http"
    "strings"
    "testing"
    "time"
)

type incompleteClockTransport struct{}
func (incompleteClockTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    return &http.Response{
        StatusCode: 200, Header: http.Header{"Content-Type": {"application/json"}},
        Body: io.NopCloser(strings.NewReader(`{"send_unix_ns":1791547200000000000}`)),
        Request: req,
    }, nil
}

func TestIncompleteJSONClockDoesNotEmitDivergence(t *testing.T) {
    p, err := probe.New(probe.Config{
        Interval: time.Second, Timeout: 200*time.Millisecond,
        Clock: func() time.Time { return time.Date(2026,10,9,12,0,0,0,time.UTC) },
        Transport: incompleteClockTransport{},
    })
    if err != nil { t.Fatal(err) }
    r := p.ProbeOnce(context.Background(), probe.Target{
        Key: "fixture", ClockURL: "http://fixture.invalid/clock",
    })
    if r.HasDivergence {
        t.Fatalf("incomplete JSON became a clock sample: %+v", r)
    }
}

Run from common: go test ./probe -run TestIncompleteJSONClockDoesNotEmitDivergence -v. The transport is a local test double; it sends no network request.

Proposed scope

Use presence-aware decoding for the JSON timestamp pair and reject incomplete pairs, preserving reachability and the existing Date fallback. Add regression cases for missing/null receive or send time, a complete pair, a usable Date fallback, and the existing explicit zero-pair behavior.

I tested a small pointer-field correction in an isolated module initialized with byte-identical copies of this package's source and existing tests. The regression suite fails on missing/null receive timestamps before the correction and passes afterward; complete-pair and missing-send controls already pass on the baseline. The correction preserves the existing explicit zero-pair fallback behavior, also covered by regression tests. All existing probe tests pass. I have not run the full repository/integration suite or changed a live node.

Could you confirm this scope is actionable and assign it to me if it is not already being handled? I would also like to know whether this bounded fix would be considered for contributor rewards; I understand a payout requires governance approval. No PR has been opened. This investigation used Codex assistance and executable regression checks.

🔄 Auto-synced from Issue #1965 every hour.