brackt/docs/infrastructure-roadmap.md

# Infrastructure Roadmap for brackt.com

## Context

brackt.com runs as a **single Node.js process** on Vultr, fronted by Traefik, backed by managed Postgres via pgbouncer.

Based on the original discovery:

- **No HA fire.** Drafts are infrequent enough that a 30-min outage isn't catastrophic *yet*.
- **Zero-downtime deploys are the immediate pain.** Connections drop on every release.
- **Low concurrency today** (<5 live drafts, <20 in 6mo) — capacity is not the constraint.
- **Avoid Redis** unless clearly justified; prefer Postgres-backed.

The roadmap sequences work by **risk-adjusted value**: cheap, reversible wins first.

---

## Current production topology

As of Phase A (merged 2026-06-08):

- **Single `brackt` container** running on Vultr, image `sjc.vultrcr.com/chrisparsons/brackt:latest`.
- **One-shot `migrate` container** runs migrations before brackt starts.
- **Traefik** fronts the app — TLS via Let's Encrypt, apex + www redirect.
- **External Postgres** via Vultr's built-in pgbouncer (`PGBOUNCER=true`).
- **No in-process `setInterval` for timers or snapshots** — draft timer is deadline-based (`picksExpiresAt`); snapshots/standings/simulations are driven by external Forgejo Actions cron jobs hitting authenticated HTTP endpoints.
- **No observability stack.** Logs live in `docker logs brackt`.
- **Still present:** `container_name: brackt` (blocks replicas), no healthcheck, `:latest` tag (painful rollbacks). These are Phase B.

---

## Roadmap

### ✅ Phase 1 — Deadline-based draft timer (complete)

The 1-second `setInterval` in `server/timer.ts` has been replaced with per-season `setTimeout` calls keyed to `picksExpiresAt` (a `timestamptz` on `draftTimers`). On process restart, active deadlines are recovered from the DB and rescheduled.

**Result:** zero per-second DB writes or socket emits during a draft. The server wakes only at pick deadlines. All callsites updated: `make-pick`, `adjust-time-bank`, draft loader, `useDraftSocketEvents`.

---

### ✅ Phase A — External HTTP cron jobs (complete, merged 2026-06-08)

Replaced the last in-process `setInterval` (`server/snapshots.ts` 24h loop) with Forgejo Actions scheduled workflows hitting authenticated endpoints.

**New endpoints** (all behind `X-Cron-Secret` header):

| Endpoint | Schedule | What it does |
|---|---|---|
| `POST /admin/jobs/run-daily-snapshots` | `5 0 * * *` | Daily fantasy standings snapshots for active/draft seasons |
| `POST /admin/jobs/sync-and-simulate` | `0 */2 * * *` | Syncs standings from external APIs; simulates only when standings changed |
| `GET /healthz` | Traefik / Docker healthcheck | 200 `{ok:true}` when DB reachable, 503 otherwise |

**Schema additions** (migration 0118): `standingsLastChangedAt`, `lastSimulatedAt` on `sportsSeasons`.

**Change detection:** `syncStandings()` compares `gamesPlayed` + `leagueRank` against current DB rows before upsert; sets `standingsLastChangedAt` only on real change. Simulation skipped if nothing changed.

**Setup required on production:** add `CRON_SECRET` to Forgejo repo secrets and to the server `.env`.

---

### 🔜 Phase B — Production hardening + deploy automation (next, ~1 day)

Three blockers for multi-replica work, plus automating compose file delivery.

#### 1. Automate compose file delivery via deploy workflow

Currently `.production-info/docker-compose-production.yml` is a manual reference — the server's `~/brackt/docker-compose.yml` drifts silently. Fix: add an `appleboy/scp-action` step to the deploy job that copies the repo's compose file to the server on every deploy.

**Deploy workflow changes:**
- Add checkout step to the deploy job (needed to access the file)
- Add SCP step: copy `.production-info/docker-compose-production.yml` → `~/brackt/docker-compose.yml`
- Build with both `:latest` and `:{sha}` tags; pass `IMAGE_TAG=${{ github.sha }}` to the SSH step
- SSH step exports `IMAGE_TAG` before running `docker compose up`

#### 2. Move secrets to a server-side `.env` file

Currently secrets reach the container via unknown means (no `env_file:` in compose, no `environment:` entries for DB credentials). Make this explicit: add `env_file: .env` to the brackt service; document all required vars in `.production-info/.env.production.example`.

Non-secret config (NODE_ENV, PGBOUNCER, APP_URL, etc.) stays inline in the compose file. Secrets (DATABASE_URL, BETTER_AUTH_SECRET, CRON_SECRET, RESEND_API_KEY, Cloudinary, Discord, Google, Sentry, Turnstile) go in the server `.env`.

#### 3. Compose file fixes

- **Remove `container_name: brackt`** — Docker rejects duplicate names; this alone blocks `replicas: 2`.
- **Add healthcheck** using `/healthz` (already shipped in Phase A):
  ```yaml
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:3000/healthz"]
    interval: 30s
    timeout: 5s
    retries: 3
    start_period: 10s
  ```
- **Pin image tag** to `${CONTAINER_REGISTRY}/brackt:${IMAGE_TAG:-latest}` — rollback becomes `IMAGE_TAG=<sha> docker compose up`.

**Files:**
- `.production-info/docker-compose-production.yml` — remove container_name, add healthcheck, parameterise image tag, add env_file
- `.production-info/.env.production.example` — template for the server-side secrets file
- `.forgejo/workflows/deploy.yml` — add checkout + SCP step, build dual tags, pass IMAGE_TAG

**Verification:**
- `curl https://brackt.com/healthz` → 200 `{ok:true}`
- Deploy with a SHA tag; confirm rollback by setting `IMAGE_TAG=<previous-sha>` and rerunning compose
- `docker compose up --scale brackt=2` no longer fails with "name already in use"

---

### 🔜 Phase C — Zero-downtime deploys via Traefik sticky sessions (~2–3 days)

Requires Phase B (no `container_name`, healthcheck, pinned tag).

Run 2 web containers behind Traefik with sticky sessions. Rolling deploy takes one replica out of rotation, upgrades it, puts it back — other replica keeps serving.

**Compose changes:**
```yaml
deploy:
  replicas: 2

labels:
  - "traefik.http.services.brackt.loadbalancer.sticky.cookie=true"
  - "traefik.http.services.brackt.loadbalancer.sticky.cookie.name=brackt_instance"
```

Socket.IO clients must return to the same instance (sticky cookie). No Redis adapter needed at current scale — the second instance is purely for deploy rotation, not capacity.

**Rolling deploy script:**
1. Scale down replica A (remove from Traefik pool)
2. Wait for drain (or 5s hard cut — reconnect works)
3. Pull new image on A, `docker compose up` A
4. Healthcheck passes → A back in pool
5. Repeat for B

**Cross-instance broadcasts** (admin action on instance B needs to reach draft clients on instance A): use a pending-broadcasts table polled every 5s by each instance. With deadline-based timers the volume is tiny. Postgres `LISTEN/NOTIFY` is the upgrade path if polling proves too slow — requires a session-mode pgbouncer pool.

**Files:** `.production-info/docker-compose-production.yml` (replicas + sticky labels), rolling deploy script.

**Verification:** rolling-deploy both instances during a staging draft; clients see at most one brief reconnect.

---

### Phase D — Optional: split tick into its own process (~1 week, not urgent)

After Phase 1, the "tick" is an idle process that wakes only at pick deadlines. It can stay co-located with web indefinitely. Only revisit if you want to deploy web without touching timer code at all.

**If/when:** extract `server/tick.ts` as a standalone entrypoint, run as a second compose service, enforce singleton via Postgres advisory lock.

---

### Phase E — Defer: Socket.IO Redis adapter

Only needed if a single draft's clients need to span multiple web instances (e.g., 50+ concurrent connections per draft, or geo-distributed). Not relevant at current scale.

If/when: add Redis, `@socket.io/redis-adapter`, drop sticky sessions. ~1-day change if Phases B–C are clean.

---

## Sequencing summary

| Phase | Status | Effort | Key win |
|---|---|---|---|
| 1 | ✅ Done | — | Deadline-based timer; eliminates 1s tick |
| A | ✅ Done | — | External cron; automated standings sync + sim; /healthz |
| B | 🔜 Next | ~1d | Compose delivered by CI; pinned tags; healthcheck; unblocks C |
| C | Blocked on B | ~2–3d | Zero-downtime rolling deploys |
| D | Optional | ~1wk | Deploy web without restarting timer process |
| E | Deferred | ~1d | Cross-instance socket fan-out (not needed yet) |

**Total to zero-downtime deploys: ~3–4 days of focused work** (Phases B + C).

---

## Open questions

- **Where do logs/metrics go?** Still unresolved. Options: structured JSON to stdout + `docker logs | jq` (zero effort), hosted (Better Stack / Axiom free tier, ~half day), or Grafana + Loki sidecars (~1 day). Pick one before adding more observability instrumentation.
- **Pgbouncer pool mode:** confirm current pool is transaction mode. Add a second session-mode pool before adopting `LISTEN/NOTIFY` or pg-boss.

---

## Critical files

- `server/timer.ts` — deadline-based scheduler; `picksExpiresAt` on `draftTimers`
- `server/socket.ts` — emits `{ expiresAt, timeRemaining }` on `timer-pick-started`
- `app/routes/healthz.ts` — DB ping; used by Docker healthcheck and Traefik
- `app/routes/admin/jobs.run-daily-snapshots.ts` — daily snapshot cron endpoint
- `app/routes/admin/jobs.sync-and-simulate.ts` — standings sync + conditional sim cron endpoint
- `app/lib/cron-auth.ts` — `requireCronSecret()` helper for cron endpoints
- `app/services/standings-sync/index.ts` — `syncStandings()` returns `{ changed: boolean }`
- `.production-info/docker-compose-production.yml` — source of truth for production compose (Phase B: delivered by CI)
- `.production-info/.env.production.example` — template for server-side secrets (Phase B)
- `.forgejo/workflows/deploy.yml` — CI/CD pipeline (Phase B: add SCP + dual image tags)
- `.forgejo/workflows/daily-snapshots.yml` — daily snapshot cron (Forgejo Actions)
- `.forgejo/workflows/sync-and-simulate.yml` — 2h standings sync + sim cron (Forgejo Actions)