Runbook

Operational knowledge for running the study stack — symptoms first.

Token expiry

Symptoms. The app suddenly shows the amber demo ribbon on screens that worked minutes ago; the API logs GCS_READ_FAILED errors (502s to the app, which then falls back to mock per-request). GET /v1/health still returns ok with "mode": "gcs" — health does not touch the bucket. If the token dies during a cold catalog build, the refresh aborts loudly ([catalog] aborting refresh: GCS auth failure (upstream 401)…) and the API keeps serving its previous catalog (or 502s on a truly cold start) — it will not serve a silently truncated family list. Beware that gcloud auth print-access-token returns gcloud's cached token: its remaining lifetime can be minutes, not the full hour.

Cause. In gcs mode with the GCS_ACCESS_TOKEN escape hatch, the token is short-lived. The data-plane ETLs treat their own tokens as stale after 45 minutes and proactively re-mint; a token pasted into the API's env does not refresh itself.

Re-arm.

# preferred: fix ADC once, drop the escape hatch entirely
gcloud auth application-default login
STUDY_DATA_MODE=gcs pnpm dev

# or re-mint the short-lived token and restart the API process
GCS_ACCESS_TOKEN=$(gcloud auth print-access-token) STUDY_DATA_MODE=gcs pnpm dev

Cloud Run deployments never use the escape hatch — reads run as the service account (scope devstorage.read_only) and do not expire this way.

Cold pack loads

Packs are fetched from GCS on first use, sha256-verified, then held in the in-process LRU (PACK_CACHE_MB, default 256 MB).

• Postflop family packs are 1.6–6.5 MB — cold hits are barely noticeable.
• Preflop depth packs are a different story: one JSON file per depth carrying the entire tree. The 100bb pack is the largest (25,832 nodes, ~120 MB on the wire). A cold first request must download, verify, and parse the whole pack — budget up to a minute for the first hit on a deep pack; every later request for that depth is served from memory.

Practical notes:

• After a deploy or restart, warm the deep packs before demoing: curl -s $API/v1/preflop/100 > /dev/null (repeat for 75/60).
• The app fetches preflop per node (/v1/preflop/{depth}/node?path=…) and caches each node response for an hour in Next's fetch cache — the pack-sized cold cost lives on the API side only, once per depth per process.
• Preflop packs have their own LRU budget, PREFLOP_CACHE_MB (default 1024). Parsed packs are charged ×4 their raw bytes (V8 heap reality: the ~122 MB 100bb pack costs ~0.5 GB resident), so the old shared 256 MB default could hold at most one deep depth and thrashed on every depth switch. Postflop keeps its separate PACK_CACHE_MB (256).

Ports

Remember: the app must point at the API via 127.0.0.1, not localhost (why).

Where data lives in GCS

Bucket gs://ollin-solver-data (project ollin-labs-496318):

backbone/berserk_v1/20260721T131354Z/    # sealed backbone read copy
  <spot>/<board>/…                       #   out.json, boundaries.jsonl.gz, logs
serving/berserk_v1/20260721T131354Z/
  v0-roots/                              # postflop root tier
    manifest.json                        #   ollin.v0_roots.manifest/1 (lands at ETL completion)
    packs/<family>.json                  #   one pack per family, sha256 in manifest
  preflop-v0/                            # preflop tier
    catalog.json
    manifest.json                        #   ollin.preflop_v0.manifest/1
    packs/<depth>bb.json                 #   one pack per depth

The sealed source of the backbone is an immutable Azure snapshot (deck/snapshots/berserk_v1/<stamp>/ in the ollinsolverdata account); the HRC preflop exports live in the Azure preflop-raw container. The serving plane only ever reads the GCS side.

How coverage refreshes

Two different mechanisms, worth keeping straight:

1. The family catalog (served coverage becoming true) — the root ETL uploads packs continuously, and the API re-lists the packs/ prefix every MANIFEST_TTL_MS (default 5 min, on access and via a background interval). New families appear without a deploy; catalog ETags change; the app's 300s revalidate picks them up. Watch progress with curl -s $API/v1/families | jq '.families | length'.
2. The coverage labels inside preflop packs — served/solved/ library/none are computed at extraction time and baked into the packs. They go stale in the optimistic direction (a boundary marked solved may in fact be served already). Refreshing the labels means re-running extract_preflop_v0.py and republishing the preflop tier — the bridge itself double-checks nothing; it trusts the pack.

Verification quick checks

curl -s $API/v1/health | jq            # mode, stamp, uptime
curl -s $API/v1/families | jq '.families | length'
curl -s "$API/v1/preflop" | jq '.depths | map({depth_bb, covered_boundaries})'
# ETag sanity: second request should be a 304
curl -s -D- -o /dev/null $API/v1/families | grep -i etag