High-availability devshard host setup

Devshard inference that stays available, backed by multiple versiond instances.

Why this matters

A single versiond process is a single point of failure (SPOF): if that machine or container dies, gateways cannot reach your host for that protocol version.

Run multiple versiond instances behind the router fleet, with shared PostgreSQL. If one instance fails, the routers send requests to the remaining instances.

Local replicas cover container failure, not machine failure. The primary host's public proxy remains a single point of failure, even with remote replicas.

Public proxy (/devshard/...)
        │
        ▼
 versiond-router fleet
        │
        ├── versiond  ──► devshardd ──┐
        ├── versiond2 ──► devshardd ──┤
        ├── versiond3 ──► devshardd ──┼── shared PostgreSQL
        │       ⋮                    │
        └── versiondN ──► devshardd ──┘

Release and images

This guide covers new HA installations, the first migration of a SQLite host to HA, and operation of HA deployments on Devshard v6.0.0.

The component images below run the host infrastructure. The protocols it serves are selected separately in VERSIOND_VERSIONS.

Edit deploy/join/config.env. Set the variables below to the shown values.

export VERSIOND_IMAGE=ghcr.io/gonka-ai/versiond:0.2.16-devshard-v6
export VERSIOND_ROUTER_IMAGE=ghcr.io/gonka-ai/versiond-router:0.2.16-devshard-v6
export PROXY_ROUTER_IMAGE=ghcr.io/gonka-ai/proxy-router:0.2.16-devshard-v6
export PROXY_POLICY_IMAGE=ghcr.io/gonka-ai/proxy:0.2.16-devshard-v6
export ORACLE_FILTER_IMAGE=python:3.12-alpine

For new installations, leave DEVSHARD_POSTGRES_IMAGE unset to use the PostgreSQL image pinned in Compose.

Prerequisites

Working node + api (dapi) + proxy on the host (standard join deployment).
Join files and host/gateway images from the same release.
Same participant identity on every HA versiond replica:
Same KEY_NAME and keyring.
Same ACCOUNT_PUBKEY.
Shared PostgreSQL; a separate data directory for each replica. Do not start a second dapi with the same keys.
Every host running a versiond replica, including remote replica hosts, needs outbound access to each peer's public address and port (PUBLIC_URL). No additional public ports are required.
Docker Compose 2.24.4+, Bash, Python 3, Git, tar, curl, jq, flock, sha256sum, timeout, xargs. Remote hosts also need ssh, psql and SSH access from the primary host to the replica host.

This guide supports local HA for protocols v4.1, v4.2 and v6.

The rolling container update and remote replica procedures below support only protocol v6. Every served protocol must be v6; protocols v4.1 and v4.2 do not provide the required shared-database check. Running v6 infrastructure images alone does not satisfy this requirement.

Install HA

Use this procedure for a new HA deployment or the first HA setup of an existing SQLite host, including hosts serving v4.1 and v4.2. Use the checkout and images from Release and images. To update an existing v6 HA deployment, use Update a v6 deployment. To add another machine, use Add a remote replica.

For an existing SQLite host, keep a private backup of config.env and every active Compose file before replacing join files or editing settings. Keep the participant identity, supported protocol list and original versiond data mount. Complete First HA setup on an existing SQLite host before starting the HA services. This migration requires downtime.

Step 1: Install PostgreSQL (preferably HA itself)

Shared PostgreSQL must also remain available. Prefer a managed or replicated database.

Choose a database

Managed PostgreSQL (recommended). Select an HA configuration and create a database and user.

Self-managed PostgreSQL. Install PostgreSQL on a dedicated host or cluster, create the role/database, and configure replication and failover for database HA.

For managed or self-managed PostgreSQL, obtain the primary endpoint, port, database, user and password. Allow connections from every replica and configure §2.2.

Local Compose PostgreSQL. docker-compose.versiond.yml starts devshard-postgres on the join host. If the machine dies, the database dies with it.

Size `max_connections`

devshardd releases for protocols v4.1 and v4.2 size their PostgreSQL connection pools by the host's CPU count, so the PostgreSQL default of 100 connections is too small for HA with these releases. Set max_connections to 250; this covers two replicas with three protocols on hosts with up to 16 CPUs. Increase it proportionally for more CPUs, replicas or protocols. For managed or self-managed PostgreSQL, set it in the server or provider settings. For local Compose PostgreSQL, the override in §2.1 sets it. The value applies only after a PostgreSQL restart.

Configure PostgreSQL credentials

Run the following command in deploy/join to save the PostgreSQL password in config.env. For a new local database, choose a password. For an existing database, enter its current password:

(
  set -euo pipefail
  read -r -s -p 'PostgreSQL password: ' DEVSHARD_POSTGRES_PASSWORD
  printf '\n'
  [[ -n "$DEVSHARD_POSTGRES_PASSWORD" ]] || exit 1
  DEVSHARD_POSTGRES_PASSWORD="$DEVSHARD_POSTGRES_PASSWORD" python3 <<'PYTHON'
import os, pathlib, re, shlex
path = pathlib.Path("config.env")
name = "DEVSHARD_POSTGRES_PASSWORD"
lines = path.read_text().splitlines()
lines = [line for line in lines if not re.match(r"^\s*(?:export\s+)?" + name + r"\s*=", line)]
lines.append("export " + name + "=" + shlex.quote(os.environ[name]))
path.chmod(0o600)
path.write_text("\n".join(lines) + "\n")
PYTHON
)

Database and user default to devshardd; override with DEVSHARD_POSTGRES_DB and DEVSHARD_POSTGRES_USER.

Local data path: ${DEVSHARD_POSTGRES_DATA_DIR:-./devshards/postgres}/data. For external PostgreSQL, complete §2.2.

Prepare the PostgreSQL directory

Local Compose PostgreSQL only. Run in deploy/join before first startup to create the data directory through Docker. The PostgreSQL entrypoint sets its ownership at startup:

source ./config.env
pg_dir="${DEVSHARD_POSTGRES_DATA_DIR:-./devshards/postgres}"
[[ "$pg_dir" = /* ]] || pg_dir="$PWD/$pg_dir"
pg_image=$(docker compose -f docker-compose.yml -f docker-compose.versiond.yml \
  config --format json | jq -er '.services["devshard-postgres"].image') &&
docker run --rm --network none --read-only \
  --security-opt label=disable \
  --volume "$pg_dir:/target:ro" \
  --entrypoint /bin/true \
  "$pg_image"

If the command fails, stop and inspect the directory permissions:

ls -ld -- "$(dirname "$pg_dir")" "$pg_dir"

Do not change database ownership manually.

Step 2: Run multiple `versiond` instances + the router fleet

Base installation files in deploy/join:

File	Purpose
`docker-compose.yml`	Base join deployment, supplied with the release
`docker-compose.versiond.yml`	Shared PostgreSQL and HA replica settings, supplied with the release
`docker-compose.devshard-ha.override.yml`	Catalog filter; create it below
`docker-compose.devshard-pg-external.override.yml`	External database settings; create it only for §2.2

2.1 Same machine, two replicas

1. Save the HA settings in config.env.

Apply Release and images first.

List approved protocol names on the node:

curl -fsS http://127.0.0.1:9100/versions | jq -er '.versions[].name'

Use the approved protocol names v4.1 and v4.2 in VERSIOND_VERSIONS. Add v6 only after it appears in the node's approved list, following Add a protocol. When updating, keep the existing supported protocol list.

Edit these variables in config.env. Keep the existing identity and database settings.

# Protocol list for a new host on this release; keep the existing list when updating.
export VERSIOND_VERSIONS="v4.1 v4.2"
# Deployment settings (retain these across component updates)
export VERSIOND_NON_HA_VERSIONS=""
# Use the same filtered catalog for replicas and both routing tiers.
export VERSIOND_ROUTING_CATALOG_URL=http://oracle-filter:9100/versions
export COMPOSE_FILE=docker-compose.yml:docker-compose.versiond.yml:docker-compose.devshard-ha.override.yml
# Append every additional override used by this deployment, in the same order.

Keep VERSIOND_NON_HA_VERSIONS empty. Include every override in COMPOSE_FILE, in the same order on every run.

2. Create docker-compose.devshard-ha.override.yml in deploy/join.

cat > docker-compose.devshard-ha.override.yml <<'EOF'
services:
  # Shared catalog of selected HA protocols for replicas and routers.
  oracle-filter:
    container_name: oracle-filter
    image: ${ORACLE_FILTER_IMAGE:?set the component images in config.env}
    environment:
      - ORACLE_UPSTREAM=http://api:9100/versions
      - ORACLE_ALLOW=${VERSIOND_VERSIONS:?set the approved HA protocol list}
      - LISTEN_PORT=9100
    command:
      - python
      - -c
      - |
        import json, os, urllib.request
        from http.server import BaseHTTPRequestHandler, HTTPServer
        UP = os.environ["ORACLE_UPSTREAM"]
        ALLOW = set(x.strip() for x in os.environ["ORACLE_ALLOW"].replace(",", " ").split() if x.strip())
        PORT = int(os.environ.get("LISTEN_PORT", "9100"))
        class H(BaseHTTPRequestHandler):
            def do_GET(self):
                if self.path.split("?",1)[0] not in ("/versions", "/"):
                    self.send_response(404); self.end_headers(); return
                with urllib.request.urlopen(UP, timeout=10) as r:
                    data = json.load(r)
                vers = [v for v in data.get("versions", []) if v.get("name") in ALLOW]
                body = json.dumps({"versions": vers}).encode()
                self.send_response(200)
                self.send_header("Content-Type", "application/json")
                self.send_header("Content-Length", str(len(body)))
                self.end_headers()
                self.wfile.write(body)
            def log_message(self, *args):
                pass
        HTTPServer(("0.0.0.0", PORT), H).serve_forever()
    depends_on:
      api:
        condition: service_started
    networks:
      default: {}
      versiond-router-back: {}
    restart: always

  # Local Compose PostgreSQL: size for every replica and protocol; see Step 1.
  devshard-postgres:
    command: ["postgres", "-c", "max_connections=250"]

  versiond:
    environment:
      - VERSIOND_ORACLE_URL=http://oracle-filter:9100/versions
      - VERSIOND_NON_HA_VERSIONS=${VERSIOND_NON_HA_VERSIONS-}
    depends_on:
      oracle-filter:
        condition: service_started
      devshard-postgres:
        condition: service_healthy

  versiond2:
    environment:
      - VERSIOND_ORACLE_URL=http://oracle-filter:9100/versions
      - VERSIOND_NON_HA_VERSIONS=${VERSIOND_NON_HA_VERSIONS-}
    depends_on:
      oracle-filter:
        condition: service_started
      devshard-postgres:
        condition: service_healthy

  # Give every extra replica the same oracle-filter environment and dependencies.

  proxy:
    environment:
      - VERSIOND_NON_HA_VERSIONS=${VERSIOND_NON_HA_VERSIONS-}
      - VERSIOND_ROUTING_CATALOG_URL=${VERSIOND_ROUTING_CATALOG_URL:?set the routing catalog URL}
      - VERSIOND_VERSIONS=${VERSIOND_VERSIONS:?set the approved HA protocol list}
EOF

2.2 External or managed PostgreSQL

If you use local Compose PostgreSQL, skip this section and go to Step 3.

Connect directly to PostgreSQL. If you use a connection pooler such as PgBouncer, configure session pooling; transaction pooling is unsupported.

This deployment does not support PostgreSQL TLS configuration through PGSSL* variables; only PGSSLMODE=disable is accepted. If your provider requires explicit TLS settings, this procedure does not apply. Do not disable required TLS.

Create the database and role through the provider, or run:

CREATE USER devshardd WITH PASSWORD '...';
CREATE DATABASE devshardd OWNER devshardd;

Allow access from every replica. Save the credentials as described in Configure PostgreSQL credentials. Create docker-compose.devshard-pg-external.override.yml with the database host and port:

services:
  devshard-postgres:
    profiles: [local-postgres]  # Leave this profile disabled.

  versiond: &external-postgres
    environment:
      PGHOST: your-managed-pg.example.com
      PGPORT: "5432"
    depends_on: !override
      api:
        condition: service_started
      oracle-filter:
        condition: service_started

  versiond2:
    <<: *external-postgres

Extra replicas: add the service name with <<: *external-postgres.

Set COMPOSE_FILE in config.env, appending any further overrides:

export COMPOSE_FILE=docker-compose.yml:docker-compose.versiond.yml:docker-compose.devshard-ha.override.yml:docker-compose.devshard-pg-external.override.yml

Step 3: Start the deployment

First HA setup on an existing SQLite host

Skip this block on a fresh host. Use it only for a single versiond that has never used PostgreSQL. Protocols v4.1 and v4.2 support this migration. Schedule maintenance outside PoC/cPoC, complete Steps 1–2, and run in deploy/join before replacing the existing container:

(
  set -euo pipefail
  umask 077
  source ./config.env
  : "${COMPOSE_FILE:?set the complete Compose file list in config.env}"
  mkdir -p backups
  backup_dir=$(mktemp -d "$PWD/backups/pre-ha.XXXXXXXX")
  echo "SQLite backup directory: $backup_dir"
  docker inspect versiond > "$backup_dir/versiond.json"
  docker compose config --format json > "$backup_dir/compose.json"
  python3 - "$backup_dir" <<'PYTHON'
import json, pathlib, sys
backup = pathlib.Path(sys.argv[1])
old = json.loads((backup / "versiond.json").read_text())[0]
new = json.loads((backup / "compose.json").read_text())["services"]["versiond"]
env = dict(item.split("=", 1) for item in old["Config"]["Env"])
if env.get("PGHOST"):
    sys.exit("This procedure is for a SQLite host that has never used PostgreSQL")
old_mounts = [(v["Type"], v["Source"]) for v in old["Mounts"]
              if v["Destination"] == "/opt/versiond/data"]
new_mounts = [(v["type"], v["source"]) for v in new["volumes"]
              if v["target"] == "/opt/versiond/data"]
if len(old_mounts) != 1 or old_mounts != new_mounts:
    sys.exit("Keep the original versiond data mount before starting HA")
PYTHON

  docker stop -t 1800 versiond
  docker cp versiond:/opt/versiond/data "$backup_dir/data"
  test -d "$backup_dir/data"
  pg_bound=$(find "$backup_dir/data" -type f -name .pg-bound -print -quit)
  if [[ -n "$pg_bound" ]]; then
    echo 'PostgreSQL-owned sessions exist; restore that database before continuing.' >&2
    exit 1
  fi

  ./versiond-router-fleet.sh prepare-networks
  # Local PostgreSQL only: explicitly allow its first initialization.
  services=$(docker compose config --services)
  if grep -qx devshard-postgres <<<"$services"; then
    DEVSHARD_POSTGRES_ALLOW_EMPTY_INIT=true \
      docker compose up -d --no-deps --wait --wait-timeout 2100 devshard-postgres
  fi
)

Downloaded versiond binaries make empty local PostgreSQL initialization fail closed. The one-time flag above confirms first HA enablement; do not save it in config.env. If PostgreSQL was used before, or a .pg-bound marker exists, preserve and recover that database instead of initializing an empty replacement. See the release's PostgreSQL storage procedure. With external PostgreSQL, use a new database dedicated to this host's replicas.

Continue below with the original versiond stopped and its data mount unchanged. The HA configuration sets DEVSHARD_STORAGE_MODE=postgres: each child imports its local SQLite sessions before serving and renames the migrated SQLite database files to *.migrated.<TS>, where <TS> is a Unix timestamp. The first replica must become ready before starting the second. Keep the backup and migrated files until the checks in Step 4 pass. Once PostgreSQL has accepted new writes, the old SQLite backup alone is no longer a current rollback copy.

Start HA

Run on the join host. Stop at the first failure:

(
  set -euo pipefail
  cd /path/to/gonka/deploy/join
  source ./config.env
  : "${COMPOSE_FILE:?set the complete Compose file list in config.env}"
  : "${VERSIOND_VERSIONS:?set the approved HA protocol list}"
  ./versiond-router-fleet.sh prepare-networks

  # Settle the database configuration before starting either replica.
  services=$(docker compose config --services)
  if grep -qx devshard-postgres <<<"$services"; then
    docker compose up -d --no-deps --wait --wait-timeout 2100 devshard-postgres
  fi
  docker compose up -d --no-deps oracle-filter
  docker compose up -d --no-deps --wait --wait-timeout 2100 versiond
  for version in $VERSIOND_VERSIONS; do
    docker exec versiond wget -qO- "http://127.0.0.1:8080/readyz?version=$version"
  done
  docker compose up -d --no-deps --wait --wait-timeout 2100 versiond2

  # Start the public listener before its private policy workers.
  docker compose up -d --no-deps proxy
  docker compose up -d --no-deps --wait --wait-timeout 2100 proxy-policy2 proxy-policy proxy

  ./versiond-router-fleet.sh apply
)

Step 4: Verify it works

4.1 Check the running services

Run on the join host after installation or an update. List every local replica in replicas; repeat the replica checks on remote hosts.

cd /path/to/gonka/deploy/join
source ./config.env
(
  set -euo pipefail
  : "${VERSIOND_VERSIONS:?set the approved HA protocol list}"
  : "${PUBLIC_URL:?set the public HTTP or HTTPS URL advertised by this host}"
  replicas=(versiond versiond2)

  ./versiond-router-fleet.sh verify-admission
  for replica in "${replicas[@]}"; do
    docker exec "$replica" wget -qO- http://127.0.0.1:8080/readyz
    for version in $VERSIOND_VERSIONS; do
      docker exec "$replica" wget -qO- "http://127.0.0.1:8080/readyz?version=$version"
    done
  done
  for version in $VERSIOND_VERSIONS; do
    ./versiond-router-fleet.sh wait-version "$version"
    curl -fsS "${PUBLIC_URL%/}/devshard/$version/healthz"
  done
)

All commands must succeed; readiness and public health endpoints must return HTTP 200. Keep the supplied /readyz Docker healthcheck.

Use PUBLIC_URL for public checks, including on HTTPS-only hosts. If the host cannot reach its own public address, run these checks from another machine.

For inference and authenticated peer checks, see Health checks pass, but inference fails.

New or replaced remote member: pass the database check before pool admission.

4.2 Check routing failover

Run after installation or a pool change, for each protocol in VERSIOND_VERSIONS. The remaining replicas must be able to handle the load.

On the public proxy host, request the protocol's health endpoint:

source ./config.env
version='<protocol from VERSIOND_VERSIONS>'
: "${PUBLIC_URL:?set the public HTTP or HTTPS URL advertised by this host}"
url="${PUBLIC_URL%/}/devshard/$version/healthz"
curl -sS -D - -o /dev/null "$url"

Expect HTTP 200. Match X-Upstream-Addr to a local container or a remote endpoint. Include every local replica:

docker inspect -f '{{.Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' versiond versiond2

Stop the replica shown in X-Upstream-Addr, on its host:

replica='<serving-container>'
docker stop -t 1800 "$replica"

On the public proxy host, repeat the same request:

curl -sS -D - -o /dev/null "$url"

Expect HTTP 200 and a different X-Upstream-Addr.

Restore the stopped replica on its host, even if the check failed:

docker start "$replica"

Pass the service checks before stopping another replica.

For routine updates, run only the service checks in §4.1.

Update a v6 deployment

Use this procedure for an HA deployment already running v6 infrastructure, with shared PostgreSQL, a router fleet, and site settings in config.env and separate Compose overrides.

Use Rolling update when every served protocol is v6 and PostgreSQL needs no restart. Use Update with downtime if the host serves v4.1 or v4.2, or if local PostgreSQL must be recreated.

Back up PostgreSQL and deployment files

Run in the existing host's deploy/join before replacing release files. Leave versiond running and save the printed backup directory.

(
  set -euo pipefail
  source ./config.env
  mkdir -p backups
  BACKUP_DIR=$(mktemp -d "$PWD/backups/pre-update.XXXXXXXX")
  printf 'Backup directory: %s\n' "$BACKUP_DIR"
  git rev-parse HEAD > "$BACKUP_DIR/previous-commit"
  docker inspect versiond > "$BACKUP_DIR/versiond.json"
  python3 - "$BACKUP_DIR" <<'PYTHON'
import json, os, pathlib, shutil, subprocess, sys
backup = pathlib.Path(sys.argv[1])
container = json.loads((backup / "versiond.json").read_text())[0]
labels = container["Config"]["Labels"]
files = [pathlib.Path("config.env").resolve()]
files += [pathlib.Path(p) for p in labels["com.docker.compose.project.config_files"].split(",")]
if os.environ.get("VERSIOND_POOL_ENDPOINTS_FILE"):
    files.append(pathlib.Path(os.environ["VERSIOND_POOL_ENDPOINTS_FILE"]).resolve())
manifest, local_files = [], []
for src in dict.fromkeys(files):
    if not src.is_file():
        sys.exit(f"Missing deployment file: {src}")
    dst = backup / "files" / str(src).lstrip("/")
    dst.parent.mkdir(parents=True, exist_ok=True)
    shutil.copy2(src, dst)
    manifest.append(str(src))
    tracked = subprocess.run(["git", "ls-files", "--error-unmatch", str(src)],
                             stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL).returncode == 0
    if not tracked or src.name == "config.env" or str(src) == str(pathlib.Path(os.environ.get("VERSIOND_POOL_ENDPOINTS_FILE", "")).resolve()):
        local_files.append(str(src))
(backup / "files.json").write_text(json.dumps(manifest, indent=2))
(backup / "local-files.json").write_text(json.dumps(local_files, indent=2))
env = dict(item.split("=", 1) for item in container["Config"]["Env"])
pg = {k: v for k, v in env.items() if k.startswith("PG") and k != "PG_POOL_MAX_CONNS"}
if not pg.get("PGHOST") or any("\n" in v or "\r" in v for v in pg.values()):
    sys.exit("Expected an existing PostgreSQL-backed versiond")
(backup / "postgres.env").write_text("".join(f"{k}={v}\n" for k, v in pg.items()))
(backup / "project").write_text(labels["com.docker.compose.project"])
PYTHON
  project=$(cat "$BACKUP_DIR/project")
  docker ps -aq --filter "label=com.docker.compose.project=$project" |
    xargs -r docker inspect > "$BACKUP_DIR/containers.json"
  if docker inspect devshard-postgres > "$BACKUP_DIR/postgres.json" 2>/dev/null; then
    [[ $(docker inspect devshard-postgres --format '{{index .Config.Labels "com.docker.compose.project"}}') == "$project" ]]
  fi
  docker pull postgres:16-alpine
  pg_args=(--rm --network container:versiond --env-file "$BACKUP_DIR/postgres.env")
  pg_major=$(docker run "${pg_args[@]}" postgres:16-alpine \
    psql -XAtw -v ON_ERROR_STOP=1 -c "SELECT current_setting('server_version_num')::int / 10000")
  [[ "$pg_major" =~ ^[1-9][0-9]*$ ]]
  pg_client="postgres:$pg_major-alpine"
  docker pull "$pg_client"
  docker run "${pg_args[@]}" "$pg_client" pg_dump -w -Fc > "$BACKUP_DIR/database.dump"
  docker run -i --rm "$pg_client" pg_restore --list < "$BACKUP_DIR/database.dump" \
    > "$BACKUP_DIR/database.contents"
  test -s "$BACKUP_DIR/database.contents"
  printf '%s\n' "$pg_client" > "$BACKUP_DIR/pg-client"
  echo 'Backup complete.'
)

Continue only after Backup complete.. The backup contains passwords; keep it private. The archive check does not test a full restore.

1. Prepare the release

Complete the backup. In deploy/join, enter its printed directory. Keep site settings in config.env and separate, untracked Compose overrides.

(
  set -euo pipefail
  read -r -p 'Pre-update backup directory: ' BACKUP_DIR
  test -d "$BACKUP_DIR"
  test -s "$BACKUP_DIR/database.contents"
  test -s "$BACKUP_DIR/previous-commit"
  git diff --binary > "$BACKUP_DIR/tracked.patch"
  git diff --cached --binary > "$BACKUP_DIR/staged.patch"
  git diff --exit-code
  git diff --cached --exit-code
  git fetch https://github.com/gonka-ai/gonka.git refs/tags/release/devshard/v6.0.0
  git switch --detach FETCH_HEAD
  # Confirm that config.env and the separate site overrides survived unchanged.
  python3 - "$BACKUP_DIR" <<'PYTHON'
import json, pathlib, sys
backup = pathlib.Path(sys.argv[1])
for name in json.loads((backup / "local-files.json").read_text()):
    path = pathlib.Path(name)
    if not path.is_file() or path.read_bytes() != (backup / "files" / name.lstrip("/")).read_bytes():
        sys.exit(f"Local configuration changed: {path}; stop before updating containers")
PYTHON
)

If tracked files have local edits, this step stops. Automatic merging is not covered; keep the saved patches and do not run git reset --hard.

Apply Release and images. Site overrides must use those image variables rather than hard-coded application images.

Before replacing services, download the application images on the machines that will run them. From deploy/join on the primary host, pull all four:

(
  set -euo pipefail
  source ./config.env
  : "${VERSIOND_IMAGE:?set the release image}"
  : "${VERSIOND_ROUTER_IMAGE:?set the release image}"
  : "${PROXY_ROUTER_IMAGE:?set the release image}"
  : "${PROXY_POLICY_IMAGE:?set the release image}"
  for image in "$VERSIOND_IMAGE" "$VERSIOND_ROUTER_IMAGE" "$PROXY_ROUTER_IMAGE" "$PROXY_POLICY_IMAGE"; do
    docker pull "$image"
  done
)

On a remote replica host, pull its configured VERSIOND_IMAGE before replacing that member. Stop if a required image cannot be downloaded. Pulling in advance ensures the updater has the intended image even if it later needs to use the local cache.

Keep the existing config.env, protocol list and Compose overrides. Add protocols separately after the update. Do not combine this update with a database move, PostgreSQL major upgrade or fleet reconfiguration.

Check max_connections for the configured replica and protocol count. Changes to the local PostgreSQL image, settings or mounts can recreate its container and disconnect every replica at once. Apply such changes with Update with downtime.

Reload and validate:

(
  set -euo pipefail
  cd /path/to/gonka/deploy/join
  source ./config.env
  # Keep every active override in the ordered COMPOSE_FILE saved in config.env.
  : "${COMPOSE_FILE:?set the complete Compose file list in config.env}"
  : "${VERSIOND_VERSIONS:?retain the approved HA protocol list in config.env}"
  docker compose config --quiet
)

2. Check the database layout

External PostgreSQL: keep the same database endpoint and credentials, leave the database running, and go to Update the deployment.

For local PostgreSQL, inspect the data directory and mounts:

docker inspect devshard-postgres --format '{{json .Mounts}}'
docker inspect devshard-postgres --format '{{json .Config.Env}}' |
  jq -r '.[] | select(startswith("PGDATA="))'

The v6 layout uses PGDATA=/var/lib/postgresql/gonka/data with ${DEVSHARD_POSTGRES_DATA_DIR:-./devshards/postgres} mounted at /var/lib/postgresql/gonka. Stop if the Compose configuration would change that host directory or its mount.

3. Update the deployment

Update one host at a time, outside PoC/cPoC. Schedule maintenance: replacing the public proxy refuses and drops connections for several seconds, and a gateway that gets a refused connection can stop sending requests to this host for up to 30 minutes.

Use the downtime procedure below if the host serves v4.1 or v4.2, or if local PostgreSQL must be recreated. Otherwise, use the rolling procedure. Leave UPDATE_SKIP_POSTGRES_PROBE and UPDATE_ACCEPT_DATABASE_CHANGE disabled.

Update with downtime

For one join host already running v6 infrastructure, using the same external database or local PostgreSQL on its existing persistent path. Keep database settings, participant identity, replica data mounts and protocols unchanged. This procedure does not cover database moves or multi-host updates.

Run in deploy/join; include every local replica in replicas. The backup contains credentials; keep its directory private.

(
  set -euo pipefail
  source ./config.env
  : "${COMPOSE_FILE:?set the complete Compose file list}"
  replicas=(versiond versiond2)
  mkdir -p backups
  backup_dir=$(mktemp -d "$PWD/backups/ha-update.XXXXXXXX")
  echo "Backup directory: $backup_dir"

  # Save the old settings and reject changes to PostgreSQL connections.
  docker inspect "${replicas[@]}" > "$backup_dir/containers.json"
  docker compose config --format json > "$backup_dir/compose.json"
  python3 - "$backup_dir" <<'PYTHON'
import json, sys
from pathlib import Path
path = Path(sys.argv[1])
model = json.loads((path / "compose.json").read_text())
reference = None
for container in json.loads((path / "containers.json").read_text()):
    service = container["Config"]["Labels"]["com.docker.compose.service"]
    old = dict(item.split("=", 1) for item in container["Config"]["Env"])
    new = model["services"][service]["environment"]
    old = {key: value for key, value in old.items() if key.startswith("PG") and key != "PG_POOL_MAX_CONNS"}
    new = {key: value for key, value in new.items() if key.startswith("PG") and key != "PG_POOL_MAX_CONNS"}
    old.setdefault("PGPORT", "5432")
    new.setdefault("PGPORT", "5432")
    if old != new or (reference is not None and old != reference):
        sys.exit(f"{service}: PostgreSQL settings differ; keep the existing connection")
    if model["networks"]["default"]["name"] not in container["NetworkSettings"]["Networks"]:
        sys.exit(f"{service}: deployment network changed; stop")
    reference = old
if not reference or any("\n" in value or "\r" in value for value in reference.values()):
    sys.exit("Cannot export PostgreSQL settings to a Docker env file")
(path / "postgres.env").write_text("".join(f"{key}={value}\n" for key, value in reference.items()))
PYTHON
  pg_network=$(jq -er '.networks.default.name' "$backup_dir/compose.json")
  pg_args=(--rm --network "$pg_network" --env-file "$backup_dir/postgres.env")

  # Select pg_dump for the running server's major version; pull before downtime.
  docker pull postgres:16-alpine
  pg_major=$(docker run "${pg_args[@]}" postgres:16-alpine \
    psql -XAtw -v ON_ERROR_STOP=1 -c "SELECT current_setting('server_version_num')::int / 10000")
  [[ "$pg_major" =~ ^[1-9][0-9]*$ ]]
  pg_client="postgres:$pg_major-alpine"
  docker pull "$pg_client"
  docker compose pull "${replicas[@]}" proxy proxy-policy proxy-policy2

  echo "Starting maintenance"
  ./versiond-router-fleet.sh stop-all --maintenance
  docker compose stop "${replicas[@]}"
  docker run "${pg_args[@]}" "$pg_client" pg_dump -w -Fc > "$backup_dir/database.dump"
  docker run -i --rm "$pg_client" pg_restore --list < "$backup_dir/database.dump" >/dev/null

  ./versiond-router-fleet.sh prepare-networks
  if [[ $(jq -r '.services.versiond.environment.PGHOST' "$backup_dir/compose.json") == devshard-postgres ]]; then
    docker compose up -d --no-deps --wait --wait-timeout 2100 devshard-postgres
  fi
  # Check capacity against the restarted PostgreSQL and its new settings.
  ./update-devshard.sh --check
  docker compose up -d --no-deps oracle-filter
  docker compose up -d --no-deps --wait --wait-timeout 2100 "${replicas[@]}"
  docker compose up -d --no-deps proxy
  docker compose up -d --no-deps --wait --wait-timeout 2100 proxy-policy2 proxy-policy proxy
  ./versiond-router-fleet.sh apply
  ./versiond-router-fleet.sh verify-admission
)

Preflight checks database access and capacity, not database identity. If an image download fails, cancel the preparation only if no container stop or replacement has occurred.

Run the service checks, then stop here.

Rolling update

This procedure updates deployment containers through update-devshard.sh and supports only protocol v6. Every served protocol must be v6, and PostgreSQL must not need a restart. If the host serves v4.1 or v4.2, or if local PostgreSQL must be recreated, use Update with downtime. Leave PostgreSQL, the filter, replicas and router fleet running.

Check every protocol on every replica:

(
  set -euo pipefail
  source ./config.env
  # Include every local member; repeat on remote hosts.
  : "${VERSIOND_VERSIONS:?set the approved HA protocol list}"
  for replica in versiond versiond2; do
    for version in $VERSIOND_VERSIONS; do
      docker exec "$replica" /bin/busybox wget -qO- -T 5 \
        "http://127.0.0.1:8080/readyz?version=$version"
    done
  done
)

Expect HTTP 200 from every check. Do not proceed if a check fails. Update one replica at a time; another replica must serve each protocol throughout the update.

Run preflight and preview the replacements. Preflight writes database probes; it does not replace services:

source ./config.env
./update-devshard.sh --dry-run

The updater replaces the router fleet, then the public proxy and nginx workers, then the versiond replicas. Keep these components on the same release and let the script manage their order.

Peer RPC may use HTTP/1.1 during the update. Complete all replacements and service checks before starting a load test. Investigate HTTP 503 responses; an unavailable HTTP/2 pool does not by itself enable fallback to HTTP/1.1.

Review the proposed images and changes. Do not proceed if preflight fails or cannot confirm that the replicas use the same database. Then run:

./update-devshard.sh

Replace remote members one at a time before final verification.

Run the service checks. To enable a new protocol, follow Add a protocol.

Add a remote replica

This procedure requires every devshardd process on the host to run protocol v6. Protocols v4.1 and v4.2 do not support the shared-database check required by this procedure.

Use a private network between machines. Do not start a second dapi with the same keys.

New deployment: finish startup and verification on the primary host first.

Machine	Runs
Primary host	Local `versiond` replicas + node/api/proxy + router fleet
Replica host	`versiond` only
Shared	PostgreSQL reachable from every `versiond` instance

Keep the new replica out of the router pool until its checks pass.

1. Prepare the primary host

On the primary host, publish these ports for the replica host on the private network:

PostgreSQL 5432, or the external database port.
Node-manager 9400.
Chain RPC/gRPC 26657, 9090.
Filtered catalog 19100.

The replica host needs outbound access to each peer's public address and port (PUBLIC_URL). Restrict inbound access to its port 8080 to the routers, as described below.

Confirm PGPASSWORD and KEYRING_PASSWORD match the running local versiond containers.

Set export GONKA_PRIVATE_BIND_IP=<primary-host-private-ip> in the primary host's config.env. Run in deploy/join:

cat > docker-compose.devshard-private.override.yml <<'EOF'
services:
  node:
    ports:
      - "${GONKA_PRIVATE_BIND_IP:?set the primary host private IP}:26657:26657"
      - "${GONKA_PRIVATE_BIND_IP:?set the primary host private IP}:9090:9090"
  api:
    ports:
      - "${GONKA_PRIVATE_BIND_IP:?set the primary host private IP}:${NODE_MANAGER_GRPC_PORT:-9400}:${NODE_MANAGER_GRPC_PORT:-9400}"
  devshard-postgres:
    ports:
      - "${GONKA_PRIVATE_BIND_IP:?set the primary host private IP}:5432:5432"
  oracle-filter:
    ports:
      - "${GONKA_PRIVATE_BIND_IP:?set the primary host private IP}:19100:9100"
EOF

External PostgreSQL: drop the devshard-postgres entry; the replica host uses the real endpoint. Append this override to the primary host's COMPOSE_FILE.

Fresh host: include the override before startup. Existing host:

Schedule maintenance for the port changes. On the primary host and each existing replica host, run in deploy/join to stop the replicas:

source ./config.env &&
mapfile -t replicas < <(docker compose ps --services | grep -E '^versiond[0-9]*$') &&
((${#replicas[@]} > 0)) &&
docker compose stop "${replicas[@]}"

On the primary host, apply the port changes:

(
  set -e
  source ./config.env
  ./versiond-router-fleet.sh stop-all --maintenance
  if docker compose config --services | grep -qx devshard-postgres; then
    docker compose up -d --no-deps --wait --wait-timeout 2100 devshard-postgres
  fi
  docker compose up -d --no-deps --wait --wait-timeout 2100 node api oracle-filter
)

On each host, restart the replicas in the same shell used to stop them:

docker compose up -d --no-deps --wait --wait-timeout 2100 "${replicas[@]}"

On the primary host, restore the fleet and run the service checks:

./versiond-router-fleet.sh apply && ./update-devshard.sh --check

2. Configure and start the replica host

For a new replica host, run on the primary host in deploy/join. Enter the replica host's SSH login as user@host. The checkout on the replica host will be created at ~/gonka, using the primary host's commit.

(
  set -euo pipefail
  source ./config.env
  read -r -p 'SSH login for the replica host (user@host): ' REPLICA_SSH
  [[ "${KEYRING_BACKEND:-file}" == file ]] || { echo "This copy procedure requires a file keyring"; exit 1; }
  ssh "$REPLICA_SSH" 'test ! -e "$HOME/gonka"'
  ssh "$REPLICA_SSH" 'git clone --no-checkout https://github.com/gonka-ai/gonka.git "$HOME/gonka"'
  release_commit=$(git rev-parse HEAD)
  ssh "$REPLICA_SSH" "git -C \"\$HOME/gonka\" checkout --detach $release_commit"
  remote_env=$(mktemp)
  trap 'rm -f "$remote_env"' EXIT
  python3 > "$remote_env" <<'PYTHON'
import os, shlex
names = ["KEY_NAME", "ACCOUNT_PUBKEY", "KEYRING_PASSWORD", "VERSIOND_VERSIONS",
         "DEVSHARD_POSTGRES_PASSWORD"]
for name in names:
    value = os.environ[name]
    if not value:
        raise SystemExit(f"Missing {name}")
    print(f"export {name}={shlex.quote(value)}")
for name, default in [("KEYRING_BACKEND", "file"), ("DEVSHARD_POSTGRES_DB", "devshardd"),
                      ("DEVSHARD_POSTGRES_USER", "devshardd"), ("NODE_MANAGER_GRPC_PORT", "9400")]:
    print(f"export {name}={shlex.quote(os.environ.get(name, default))}")
print('export VERSIOND_NON_HA_VERSIONS=""')
if os.environ.get("VERSIOND_IMAGE"):
    print("export VERSIOND_IMAGE=" + shlex.quote(os.environ["VERSIOND_IMAGE"]))
PYTHON
  ssh "$REPLICA_SSH" 'touch "$HOME/gonka/deploy/join/config.env" && chmod 600 "$HOME/gonka/deploy/join/config.env" && cat > "$HOME/gonka/deploy/join/config.env"' < "$remote_env"
  docker exec versiond /bin/busybox tar -C /root/.inference -cf - keyring-file |
    ssh "$REPLICA_SSH" 'mkdir -p "$HOME/gonka/deploy/join/.inference"; tar --no-same-owner -xf - -C "$HOME/gonka/deploy/join/.inference"'
)

On the replica host, run cd ~/gonka/deploy/join. For an existing replica host, use Replace a member.

Add to the replica host's config.env, with the real private addresses:

export NETWORK_NODE_PRIVATE_IP='<primary-host-private-ip>'
export VERSIOND_BIND_IP='<replica-host-private-ip>'
export COMPOSE_FILE=docker-compose.versiond-remote.yml:docker-compose.versiond-remote-filter.yml

External PostgreSQL: also set DEVSHARD_POSTGRES_HOST and DEVSHARD_POSTGRES_PORT. Restrict the replica host's port 8080 to the routers.

Create the filter override in the replica host's deploy/join:

cat > docker-compose.versiond-remote-filter.yml <<'EOF'
services:
  versiond:
    environment:
      VERSIOND_ORACLE_URL: http://${NETWORK_NODE_PRIVATE_IP:?set the primary host private IP}:19100/versions
EOF

Start the remote replica and check every selected protocol:

source ./config.env
docker compose up -d --wait --wait-timeout 2100
(
  set -e
  : "${VERSIOND_VERSIONS:?set the approved HA protocol list}"
  for version in $VERSIOND_VERSIONS; do
    curl -fsS "http://${VERSIOND_BIND_IP}:8080/readyz?version=$version"
  done
)

Continue when the container is healthy and every check returns HTTP 200.

Check the remote database

Check the replica host's database before adding the replica to the router pool. Run both blocks below on the replica host; replace /path/to/gonka with its checkout directory (~/gonka for a host created above):

(
  set -euo pipefail
  cd /path/to/gonka/deploy/join
  command -v psql
  test ! -e pool-postgres.env || exit 1
  cp pool-postgres.env.template pool-postgres.env
  chmod 600 pool-postgres.env
)

Fill in PGHOST, PGPORT, PGDATABASE, PGUSER and PGPASSWORD in the checkout's deploy/join/pool-postgres.env using the primary host's database settings or values from the database administrator. Run:

(
  set -euo pipefail
  cd /path/to/gonka/deploy/join
  ./update-devshard.sh --check-storage --reference-env ./pool-postgres.env
)

Expect Storage check passed and exit code 0. The check writes test data to the database; run it separately from other checks and updates. For additional containers, repeat --container NAME.

3. Add the replica to the router pool

On the primary host, create versiond-endpoints.json in deploy/join. Replace 10.0.0.12 with the replica host's private IP and include every replica. All routers must be able to connect to these addresses:

cat > versiond-endpoints.json <<'EOF'
[
  {"id": "local-a", "host": "versiond", "port": 8080},
  {"id": "local-a-2", "host": "versiond2", "port": 8080},
  {"id": "remote-b", "host": "10.0.0.12", "port": 8080}
]
EOF

Set export VERSIOND_POOL_ENDPOINTS_FILE=./versiond-endpoints.json in the primary host's config.env; source ./config.env. Fresh fleet: run ./versiond-router-fleet.sh apply. Existing fleet: run during maintenance:

VERSIOND_ROUTER_ALLOW_MAINTENANCE_OUTAGE=true \
  ./versiond-router-fleet.sh maintenance-rollout
./versiond-router-fleet.sh verify-admission

The rollout interrupts new requests. Keep member IDs when replacing a host at the same endpoint. Endpoint-file edits take effect only through the rollout.

Run the service checks and routing failover check after adding the replica.

Add a local replica

After startup and verification, add versiond3:

In config.env, insert docker-compose.versiond3.yml immediately after docker-compose.versiond.yml in COMPOSE_FILE. Keep the filter, database and site overrides after it:

export COMPOSE_FILE=docker-compose.yml:docker-compose.versiond.yml:docker-compose.versiond3.yml:docker-compose.devshard-ha.override.yml
# External PostgreSQL: append :docker-compose.devshard-pg-external.override.yml
# Retain every other site override after these files.

Add under services in docker-compose.devshard-ha.override.yml:

versiond3:
  environment:
    VERSIOND_ORACLE_URL: http://oracle-filter:9100/versions
    VERSIOND_NON_HA_VERSIONS: ${VERSIOND_NON_HA_VERSIONS-}
  depends_on:
    oracle-filter:
      condition: service_started
    devshard-postgres:
      condition: service_healthy

External PostgreSQL: also add under services in docker-compose.devshard-pg-external.override.yml, below the existing anchor:

versiond3:
  <<: *external-postgres

Local PostgreSQL: run Update with downtime for the existing replicas to apply the new PostgreSQL mount. Skip this step for an external database.
Start the new replica:

source ./config.env
docker compose config --quiet
docker compose pull versiond3
docker compose up -d --no-deps --wait --wait-timeout 2100 versiond3

If using versiond-endpoints.json, add versiond3 and apply the updated list. Default Docker discovery finds it automatically. Run the service checks, including versiond3.

For a fourth replica, copy docker-compose.versiond3.yml, replace 3 with 4, and repeat the overrides above.

Operate the deployment

Each versiond replica is a member of the router pool.

Run the commands below from deploy/join. Manage routers on the public proxy host and versiond replicas on the host running the replica.

Manage routers

Use versiond-router-fleet.sh to manage routers. The script reads config.env and runs each router in a separate Docker Compose project. Running docker compose down for the main join deployment therefore leaves the routers running.

To stop all routers, run ./versiond-router-fleet.sh stop-all --maintenance. The default shutdown timeout is 30 minutes per router (VERSIOND_ROUTER_DRAIN_TIMEOUT_SECONDS).

Each router instance occupies a numbered slot. The slot number stays the same when its container is replaced. Use this number in fleet commands:

Task	Command
View router status and slot numbers	`./versiond-router-fleet.sh status`
Stop the router in slot 0	`./versiond-router-fleet.sh stop 0`
Start the router in slot 0	`./versiond-router-fleet.sh start 0`
Verify routing after a change	`./versiond-router-fleet.sh verify-admission`
Apply router image and configuration changes	`./versiond-router-fleet.sh apply`

Full release update: follow Update the deployment.

After an interrupted operation, keep the previous stopped containers and catalog volumes; rerun with the same image and configuration. For pool or resolver changes, follow membership maintenance.

Before machine maintenance, stop the routers gracefully, then stop the main stack:

(
  set -euo pipefail
  ./versiond-router-fleet.sh stop-all --maintenance
  # Stop the main stack after the fleet has drained.
  source ./config.env || exit 1
  : "${COMPOSE_FILE:?set the complete Compose file list}"
  docker compose stop
  # Only when decommissioning, after the main stack is down:
  # ./versiond-router-fleet.sh down --maintenance
)

Restart a member

Restart one replica at a time. The remaining replicas must serve all protocols and handle the load. Include every active override in COMPOSE_FILE:

(
  set -euo pipefail
  cd /path/to/gonka/deploy/join
  source ./config.env
  # COMPOSE_FILE must include every active override, including external PG,
  # private ports and additional replicas; keep this list in config.env.
  : "${COMPOSE_FILE:?set the complete Compose file list in config.env}"
  dc=(docker compose)

  # Stop only one member, keeping enough other replicas ready to handle the load.
  "${dc[@]}" stop versiond2
  # Restart the same member with its existing data:
  "${dc[@]}" up -d --no-deps --wait --wait-timeout 2100 versiond2
  (
  set -e
  : "${VERSIOND_VERSIONS:?set the approved HA protocol list}"
  for version in $VERSIOND_VERSIONS; do
    docker exec versiond2 wget -qO- "http://127.0.0.1:8080/readyz?version=$version"
  done
  )
)

Run the readiness checks above before restarting another replica.

Replace a member

Use this procedure to replace a replica in an existing v6 deployment serving only protocol v6. If the host serves v4.1 or v4.2, use Update with downtime.

Keep the same database, participant identity, protocol list and data mounts. Replace one replica at a time; another replica must serve each protocol.

On the host running the replica, run in its existing deploy/join directory to fetch and check out the release:

(
  set -euo pipefail
  git diff --exit-code
  git diff --cached --exit-code
  git fetch https://github.com/gonka-ai/gonka.git refs/tags/release/devshard/v6.0.0
  git switch --detach FETCH_HEAD
)

If Git reports local changes or conflicting files, stop; do not force checkout. Keep config.env and site overrides in separate, untracked files. After checkout succeeds, apply the variables in Release and images. 2. For a remote member, remove its entry from versiond-endpoints.json on the primary host and run membership maintenance. On the member's machine, run in deploy/join to select and replace the replica:

(
  set -euo pipefail
  source ./config.env
  docker compose ps --services
  read -r -p 'Replica service from the list (for example versiond2): ' service
  [[ "$service" =~ ^versiond[0-9]*$ ]] || exit 1
  docker compose stop "$service"
  docker compose pull "$service" &&
  docker compose up -d --no-deps --wait --wait-timeout 2100 "$service"
)

For a remote member, pass the database check before restoring its entry in the primary host's endpoint file and applying membership maintenance. 3. Pass the service checks before replacing the next member. If checks fail, stop before replacing another member. Reverting an image does not revert database changes; any rollback version must support the current database.

Remove a member

On the replica's host, run in deploy/join:

(
  set -euo pipefail
  source ./config.env
  docker compose ps --services
  read -r -p 'Replica service to remove (for example versiond2): ' service
  [[ "$service" =~ ^versiond[0-9]*$ ]] || exit 1
  docker compose stop "$service" && docker compose rm -f "$service"
)

Edit config.env. To disable versiond2, set:

export VERSIOND2_REPLICAS=0

For an extra replica, remove its Compose filename from COMPOSE_FILE and its service block from the filter/database overrides. Keep its data directories. If using versiond-endpoints.json, remove its entry on the primary host and run membership maintenance. Run the service checks for the remaining replicas.

Add a protocol

Check the release guide for the host/gateway versions. Once it appears in the node's approved protocol list, add it to VERSIOND_VERSIONS in config.env on every host. Keep existing protocols.
On the primary host: source ./config.env, then docker compose up -d --no-deps oracle-filter with the complete COMPOSE_FILE.
./versiond-router-fleet.sh wait-version <new-protocol>, then run the service checks with the updated list.

No router restart is needed. The next host update applies the saved protocol list to the router fleet and public proxy; schedule it as maintenance.

Keep proxy-router-state and the router-state volume for every router slot. Remove a protocol only during maintenance, after its sessions are no longer needed: a filter change can stop children, while accepted router routes persist by default.

Troubleshooting

Health checks pass, but inference fails

/healthz, /readyz, verify-admission and update-devshard.sh --check do not send inference requests or authenticate a peer.

Send a gateway test request through an escrow containing this host. Use a different message each time, such as one containing the current timestamp, to avoid the gateway response cache. Confirm in the host's logs that it served the request.
Repeat with one replica stopped, following the routing failover check, then restore the replica even if the request failed. Expect a model response and requests reaching the remaining replica.
For v6 and later, confirm that an existing peer completes Attach, keeps Watch open, completes a subsequent RPC and renews its session. On the peer, devshard_peer_attach_total{result="ok"} should increase and devshard_peer_session_state{state="ready"} should stay at 1 for this host and version. Inspect peer and versiond logs for authentication or session errors; see the release's peer RPC checks.

After an explicit HTTP/2 protocol mismatch, a peer retries discovery in about 30 minutes; an ambiguous connection closure starts a 30-second exponential backoff. Working HTTP/1.1 sessions during discovery are expected.

Leave DEVSHARD_RPC_GRPC unset to use the default peer transport selection.

Cancel before maintenance

Use this only if preparation failed before any container stop or replacement. This restores files only. Run in deploy/join; use the Backup directory printed by the pre-update backup:

read -r -p 'Pre-update backup directory: ' BACKUP_DIR
export BACKUP_DIR
(
  set -euo pipefail
  test -s "$BACKUP_DIR/previous-commit"
  git switch --detach "$(cat "$BACKUP_DIR/previous-commit")"
  python3 - "$BACKUP_DIR" <<'PYTHON'
import json, pathlib, shutil, sys
backup = pathlib.Path(sys.argv[1])
for name in json.loads((backup / "files.json").read_text()):
    target = pathlib.Path(name)
    source = backup / "files" / name.lstrip("/")
    if not source.is_file():
        sys.exit(f"Missing backup file: {source}")
    shutil.copy2(source, target)
PYTHON
)

If git switch refuses because of local changes, stop and preserve those changes; do not force checkout or run git reset --hard.

After a successful restore, open a new terminal or SSH login session, return to deploy/join, and run:

source ./config.env

Do not reuse the update shell: it may retain exported settings absent from the restored file. Starting another Bash inside that shell also inherits those settings. Keep the backup. For denied or manifest unknown during image download, wait for access to the release images; do not substitute another tag.

Resume an interrupted rolling update

If the updater reports pending recovery, rerun it in deploy/join. Use the same OS user, release and configuration, including any UPDATE_STATE_DIR or XDG_STATE_HOME setting. Do not switch between sudo and non-sudo runs. Keep the recovery directory printed by the updater. Run without --check or --dry-run to recover and continue the update:

source ./config.env
./update-devshard.sh

Resolve a missing database

Stop before starting replicas against an empty database. For a local database, inspect the container and its mounts:

docker compose logs --tail=100 devshard-postgres
docker inspect devshard-postgres --format '{{json .Mounts}}' | jq .

If the container has been removed, inspect its saved mounts from the pre-update backup:

(
  set -euo pipefail
  read -r -p 'Pre-update backup directory: ' backup_dir
  test -s "$backup_dir/postgres.json"
  jq -e '.[0].Mounts' "$backup_dir/postgres.json"
)

Compare the mounts with the v6 database layout and the saved configuration. If the host data directory is missing or empty, keep replicas stopped and recover the database from backup before restarting them. Database restore procedures depend on the PostgreSQL deployment and are outside this guide. Do not use DEVSHARD_POSTGRES_ALLOW_EMPTY_INIT to bypass recovery.