Skip to content

devshard: GATEWAY_LOGPROBS_OPTIMIZATION_OVERRIDE is ignored after the gateway's first start #2016

Open @yaUrich opened 2026-10-15 10:51 UTC 2 comments Updated 2026-10-15 20:36 UTC

Summary

GATEWAY_LOGPROBS_OPTIMIZATION_OVERRIDE takes effect only on the gateway's first start. After that the gateway reads the setting from its DB, so a new value in the env is ignored on restart. The admin API (POST /v1/admin/settings with logprobs_optimization) works at runtime and is not affected.

Why

An operator who changes the env and restarts the gateway keeps the old value. A small follow-up could make the env apply on restart too.

Details

  • The env is read into bootstrapSettings (main.go#L254).
  • Initialize writes them only when the settings row is missing (Postgres, SQLite).

Reproduced on testenv at d1b9df2:

  1. Set {"logprobs_optimization":{"enabled":false}} through the admin API, then restart the gateway. Result: false.
  2. Recreate the gateway with GATEWAY_LOGPROBS_OPTIMIZATION_OVERRIDE=true. Result: still false.
  3. Start a second gateway on an empty DB with the env set to false. Result: false, so the env is applied.

Found while testing the v6 logprobs switch: paranjko/external-test-lab#246.

💬 Comments (2)

@redstartechno commented 2026-10-15 13:42 UTC

Taking this one: a small change in mustBootstrapGatewayState that re-applies an explicitly set GATEWAY_LOGPROBS_OPTIMIZATION_OVERRIDE after Initialize, with a test on both store backends. PR follows in a minute.

@redstartechno commented 2026-10-15 20:36 UTC

Follow-up: the PR for this (#2024) was closed as working by design. Once the gateway state exists, the stored value has priority over GATEWAY_LOGPROBS_OPTIMIZATION_OVERRIDE, so the env var only seeds the first bootstrap and a changed value on restart is ignored on purpose.

To change the setting on a running gateway, use the admin endpoint documented in docs/proposal/compressed-payloads.md (it persists through store.UpdateSettings):

POST /v1/admin/settings  {"logprobs_optimization": {"enabled": false}}
POST /v1/admin/settings  {"logprobs_optimization": {}}   # clear, defer to the hosts again

So this is at most a documentation gap (the env var table in that document does not say the variable is bootstrap-only), not a code change.

🔄 Auto-synced from Issue #2016 every hour.