brackt/docs/infrastructure-roadmap.md

# Infrastructure Roadmap for brackt.com

## Context

brackt.com today runs as a **single Node.js process** that bundles three concerns:

1. The Express + React Router 7 SSR web server (`server.ts`)
2. A 1-second `setInterval` tick loop driving draft timers and auto-picks (`server/timer.ts`)
3. A Socket.IO server with **in-memory** room/connection state (`server/socket.ts`)

Plus a 24h snapshot interval (`server/snapshots.ts`). There is no job queue, no scheduler beyond `setInterval`, and no pub/sub layer. Deploys drop all socket connections; a crash takes down drafts, web, and the tick simultaneously.

Based on the discovery questions:

- **No HA fire.** Drafts are infrequent enough that a 30-min outage isn't catastrophic *yet*.
- **Zero-downtime deploys are the immediate pain.** Connections drop on every release.
- **Low concurrency today** (<5 live drafts, <20 in 6mo) — capacity is not the constraint.
- **Avoid Redis** unless clearly justified; prefer Postgres-backed.
- **Go rewrite is speculative.** No measurements show Node is the bottleneck.

The roadmap below sequences work by **risk-adjusted value**: cheap, reversible wins first; expensive rewrites only when data demands them.

---

## Current production topology

What's actually deployed today (per `.production-info/docker-compose-production.yml`):

- **Single `brackt` container** running on Vultr, image pulled from `sjc.vultrcr.com/chrisparsons/brackt:latest`.
- **One-shot `migrate` container** runs `node /app/scripts/migrate.mjs` before brackt starts (`depends_on: condition: service_completed_successfully`).
- **Traefik** is already fronting the app — TLS via Let's Encrypt, `brackt.com` apex + `www` → apex redirect, port 3000.
- **External Postgres** (not in this compose). `PGBOUNCER=true` is set in env — the app connects through Vultr's built-in pgbouncer. Vultr supports per-pool mode selection (session/transaction/statement), so worst case is "add a second pool" rather than re-architecting.
- **No observability stack.** No Prometheus, Loki, Grafana, no log shipper. Logs live in `docker logs brackt`.
- **No Redis, no separate worker, no tick container.** Everything is one process.
- `restart_policy: condition: on-failure, max_attempts: 5` — after 5 crashes the container stays down with no alerting.
- **Hidden blockers for any multi-replica future:** `container_name: brackt` is hardcoded (Docker won't allow two containers with the same name), no healthcheck on the brackt service, and the image tag is `:latest` (rollbacks are painful and you can't tell which replica is on which version mid-deploy).

These facts shape the phases below — particularly the deadline-based timer recommendation that sidesteps several of these constraints entirely.

---

## Roadmap

### Phase 0 — Instrumentation (1–2 days) ⭐ do this first

You cannot make further sequencing decisions without data. Almost everything below ("is the tick slow?", "do we need multi-instance?", "is the timer refactor working?") becomes guesswork without basic metrics.

**Add:**
- Tick drift measurement in `server/timer.ts` — log when actual interval > 1100ms.
- Auto-pick latency: time from `timeRemaining <= 0` → `executeAutoPick()` resolved.
- Concurrent socket connections + active draft rooms gauge.
- Basic HTTP request duration histogram (any lightweight middleware).

**Where to look at the data — pick one, don't leave this abstract:**

| Option | Effort | What you get |
|---|---|---|
| Structured JSON logs to stdout, view via `docker logs brackt \| jq` | ~2 hours | Good enough for low-volume signals (tick drift, auto-pick latency). Start here. |
| Grafana + Loki + Promtail as sidecars in the compose | ~1 day | Real dashboards, log retention, query history |
| Hosted (Axiom / Better Stack / Grafana Cloud free tier) | ~half day | No infra to run; ship logs over the wire |

**Recommendation:** start with structured JSON + `jq` for a week, escalate only if you find yourself wanting historical queries you can't get from logs.

**Why first:** every decision below is currently a guess. A week of metrics turns guesses into decisions, *and* gives the Phase 1 timer refactor a baseline to compare against.

---

### Phase 1 — Deadline-based draft timer (3–5 days) ⭐ biggest win

**The problem:** the 1-second `setInterval` in `server/timer.ts` is the *cause* of nearly every downstream problem in this roadmap. Per active draft, every second it issues an UPDATE on `draftTimers`, several SELECTs, and a `timer-update` socket broadcast. It also makes the process un-killable without dropping drafts, makes multi-instance hard, and is the only thing in this stack that looks like it might need Go.

**The insight:** the timer state is *deterministic from a deadline*. If you store `pickDeadlineAt` (absolute UTC) instead of `timeRemaining` (seconds), the remaining time at any moment is just `pickDeadlineAt − now()`. The server doesn't need to tick — it only needs to wake up *at the deadline* to run the auto-pick.

**The change:**

Schema (one new column total):
```
seasons
  + pickDeadlineAt timestamptz NULL   -- the active deadline for the current pick;
                                      -- NULL while paused, pre-draft, or completed
```
Why on `seasons` and not `draftTimers`: only one team's deadline matters at a time (the current picker's). One row per draft matches the semantics and gives a trivial bootstrap query (`SELECT id FROM seasons WHERE status='draft' AND pickDeadlineAt IS NOT NULL`). The existing `draftTimers.timeRemaining` stays where it is — still needed for chess-clock bank tracking and as the "saved remaining seconds while paused" storage.

Server behavior — funnel **every** deadline write through a single helper:
```ts
async function setNextPickDeadline(tx, { seasonId, teamId, budgetSeconds }) {
  // 1. UPDATE seasons SET pickDeadlineAt = now() + budgetSeconds
  // 2. UPDATE draftTimers SET timeRemaining = budgetSeconds (chess-clock: add increment)
  // 3. (re)schedule in-memory setTimeout for this seasonId
  // 4. emit { deadline, serverNow } on draft-${seasonId}
}
```
Callsites (audit surface):
- **Draft auto-start** (`app/services/draft-autostart.ts`) — first pick's deadline.
- **Manual pick** (`app/routes/api/draft.make-pick.ts`) — next pick's deadline (currently writes `timeRemaining` only).
- **Auto-pick** (`server/timer.ts` rewrite) — next pick's deadline after auto-picking.
- **Admin time-bank adjust** (`app/routes/api/draft.adjust-time-bank.ts`) — recompute `pickDeadlineAt` to match new `timeRemaining`.
- **Resume from pause** — same as auto-start but for the current picker.

Other behaviors:
- **Process restart:** on boot, query active drafts where `pickDeadlineAt IS NOT NULL` and re-schedule a `setTimeout` for each. Already-expired deadlines auto-pick immediately.
- **Pause:** save remaining into existing `draftTimers.timeRemaining` (`= pickDeadlineAt − now()`), null out `seasons.pickDeadlineAt`, clear the in-memory timeout. **No new "paused remaining" column needed** — `timeRemaining` already serves this purpose.
- **Overnight pause:** when scheduling, if the deadline falls inside the pause window, schedule for `resumeAtUTC` instead and emit a `frozen` flag.
- **`while_on` autodraft:** unchanged semantics — fire immediately on pick transition, capped at `totalTeams` iterations. The logic moves from the tick to the pick-completion handler (inside `setNextPickDeadline` or just before it).

Client behavior:
- Receive `{ deadline, serverNow }`; compute clock offset once.
- Render countdown locally from `deadline − clientNow() + offset`. Already has to do this between 1Hz emits today, so the interpolation code likely exists.

**Wins vs today:**

| | 1s setInterval (today) | Deadline-based |
|---|---|---|
| DB writes per active draft per pick | ~30–120 | 1 |
| Socket emits per active draft per pick | ~30–120 | 1 (deadline) + 1 (pick made) |
| Server wake-ups per second | 1 always | 0 (sleeps until deadline) |
| Survives process restart | dies | one-line bootstrap query |
| Cross-process / multi-instance | hard (singleton tick required) | trivial (whichever instance handles the pick schedules the next deadline) |

**Critical constraints:**
- **Single scheduler assumption.** Phase 1 runs in one process. Multi-instance leader-election (per-`seasonId` advisory lock + claim metadata) is deferred to Phase 5. Document this assumption explicitly in `server/timer.ts`.
- **Clock skew.** Send `serverNow` with each deadline event so clients can correct.
- **Reconnect path.** The loader must return `pickDeadlineAt` (and `serverNow`) so clients re-initializing after a reconnect render the correct countdown without waiting for a socket event.
- **Migration safety.** Run both systems in parallel for one draft (deadline-based writes the new column; old setInterval still drives picks). Compare. Flip when confident.

**Why this matters more than splitting the tick into its own process:** it dissolves the problem rather than relocating it. Once the tick is deadline-based and idle 99% of the time, splitting it into its own container becomes a deploy-isolation play, not a correctness one. And the Phase 5 Go rewrite goes off the table permanently — there is no hot loop left to optimize.

**Files:** `server/timer.ts` (rewrite as a scheduler module), `app/database/schema.ts` + new migration (single column on `seasons`), `app/models/draft-timer.ts` (new `setNextPickDeadline` helper — see implementation guide), `app/services/draft-autostart.ts` (call helper on draft start), `app/routes/api/draft.make-pick.ts` (call helper on manual pick), `app/routes/api/draft.adjust-time-bank.ts` (recompute deadline on adjust), `app/hooks/useDraftSocketEvents.ts` (sole `timer-update` consumer — switch to deadline event), draft room loader (`app/routes/leagues/$leagueId.draft.$seasonId.tsx`), socket event types.

**See [`docs/agents/deadline-based-timer.md`](agents/deadline-based-timer.md) for the implementation guide** — schema, helper signature, callsite-by-callsite migration, test plan.

**Gotchas to handle (load-bearing for correctness):**

1. **Stale in-memory `setTimeout` after out-of-band DB writes.** Anything that mutates `pickDeadlineAt` outside the scheduler (admin extend, retro edit) leaves the captured closure stale. *Mitigation:* the auto-pick callback must re-read the row and verify `pickDeadlineAt` still equals the expected value before firing; if changed, reschedule and return. This single guard also handles wall-clock vs. monotonic drift (NTP jumps, VM resume): the re-check catches "deadline already passed" and "deadline moved" with the same code path.
2. **Pause-during-flight race.** `setTimeout` fires → enters auto-pick logic → admin pauses concurrently. The re-read in (1) must also check `draftPaused` before proceeding. Contract: in-flight picks complete; future scheduled picks don't fire after pause.
3. **Idempotency on restart.** Bootstrap query (`pickDeadlineAt IS NOT NULL`) fires all expired deadlines on boot. Idempotency is **already guaranteed** by `executeAutoPick` (`app/models/draft-utils.ts:499–512` pre-check + `:629–650` `ON CONFLICT DO NOTHING` on `(seasonId, pickNumber)`). Add an explicit test: "auto-pick is idempotent across process restart."
4. **Transaction boundary.** Pick row + next `pickDeadlineAt` write must share one transaction. Otherwise a crash between them leaves the draft "completed pick / no deadline" — the bootstrap query won't catch it.

**Lower-severity edge cases (worth knowing, not blocking):**

- **`setTimeout` 24.8-day overflow:** irrelevant for per-pick deadlines; *don't* reuse this primitive for `season.draftDateTime` auto-start — keep that on a low-frequency (30–60s) `setInterval` poll.
- **`while_on` cascade** moves from the tick to the pick-completion handler: check next team's autodraft mode before scheduling next deadline; fire immediately if `while_on`, capped at `totalTeams` iterations.
- **Client tab backgrounding** is *better* than today: browsers throttle timers, but `visibilitychange → visible` triggers a recompute from the current deadline, so countdowns snap to truth instead of waiting for the next server emit.
- **Clock skew:** server includes `serverNow` with every deadline event; client computes a one-time offset.
- **Migration of in-flight drafts:** two-stage deploy — first ship shadow writes alongside the existing tick (observe a day), then flip the authoritative path and drop the interval.
- **Multi-instance scheduling** (Phase 5 inheritance): defer the leader-election design; document the "single scheduler" assumption explicitly. The bootstrap query recovers a crashed scheduler on restart, which is sufficient for one instance.
- **Observability swap:** the Phase 0 "tick drift" metric becomes meaningless after Phase 1. Replace with **deadline-fire accuracy** = `actualFireTime − scheduledFireTime`. Same intent.

---

### Phase 2 — Scheduled work: snapshots off `setInterval` (1 day)

**The problem:** `server/snapshots.ts` uses a 24h `setInterval` that dies with the process, has no retry, and no visibility. As you add standings sync, scoring imports, EV snapshots, etc., this gets worse.

**The realistic options** (the original roadmap assumed pg-boss; given the production constraints, simpler choices win here):

| Option | Pros | Cons | Verdict |
|---|---|---|---|
| **External HTTP cron** (healthchecks.io free / GitHub Actions `schedule:` / cron-job.org) hitting `POST /admin/jobs/run-daily-snapshots` | Zero new infra; dead-man-switch alerting built-in; sidesteps pgbouncer entirely | External dependency; needs auth on the endpoint | ⭐ start here |
| `node-cron` in a small worker container with Postgres advisory lock for singleton | Self-contained; trivial; no 3rd-party | Build basic retry/durability yourself; another container | Good middle ground |
| `pg-boss` | Real job system, retries, audit table | Uses `LISTEN/NOTIFY` and prepared statements — incompatible with pgbouncer transaction pooling. **On Vultr, this is solvable:** create a second session-mode pool just for pg-boss and pass its DSN to that workload only. No built-in UI. | Defer until you have 5+ scheduled jobs |
| `pg_cron` extension | Runs inside Postgres; rock solid; no scheduler process. **Supported on Vultr** (`CREATE EXTENSION IF NOT EXISTS pg_cron;`). | Logic lives in SQL; for app-level work you'd call out via `pg_net → POST /admin/jobs/snapshots`, which is just external HTTP cron with extra steps | Skip for one job; reconsider if you accumulate many |

**Recommendation:** external HTTP cron. The "job" becomes an authenticated `POST /admin/jobs/run-daily-snapshots` route that calls the existing `createSnapshotsForAllSeasons()`. Healthchecks.io's free tier gives you "you didn't run today" alerts for free.

**Why now (not later):** it's a 1-hour change once you have an admin endpoint, and it removes the last `setInterval` from `server.ts` — which makes Phase 3 (process restart safety) trivially correct rather than "hopefully correct."

**Files:** new admin route `app/routes/admin/jobs.run-daily-snapshots.tsx` (POST handler with shared-secret auth), remove `startSnapshotSystem()` call from `server.ts`, delete or repurpose `server/snapshots.ts`.

---

### Phase 3 — Production hardening prerequisites (1–2 days)

Before any multi-replica / zero-downtime work, the production compose needs three small fixes. These are cheap and worth doing regardless of what comes next.

**The changes:**
1. **Remove hardcoded `container_name: brackt`.** Docker rejects duplicate names — this alone blocks scaling to 2 replicas. Use the service name + replica index (Compose handles this automatically when `container_name` is omitted).
2. **Add a healthcheck** to the brackt service (e.g., `GET /healthz` returns 200 once the server is listening and DB is reachable). Without this, Traefik will route to a half-started container during rolling deploys.
3. **Pin the image tag.** Move off `:latest` to a Git SHA or version. Rollbacks become a one-line edit; you can tell which replica is running which version during a rolling deploy.

**Files:** `.production-info/docker-compose-production.yml`, new `app/routes/healthz.tsx`, deploy script (whatever sets the tag).

---

### Phase 4 — Optional: split tick into its own process (1 week, no longer urgent)

**Status after Phase 1:** the "tick" is now an idle process that wakes up at deadlines. It can be co-located with web indefinitely without correctness or performance issues.

**When to revisit:** if you want pure deploy isolation — restart the web tier without touching the timer scheduler at all. This is a nice-to-have, not a blocker.

**The change (if/when):**
- Extract timer scheduling into `server/tick.ts` standalone entrypoint.
- Run as a second container in compose.
- Tick still emits via Socket.IO — either via Postgres `LISTEN/NOTIFY` to the web tier (⚠️ verify pgbouncer mode supports session-scoped listeners) or via a pending-broadcasts table the web tier polls. With Phase 1 done, the broadcast volume is tiny — polling is fine.
- Singleton enforced via Postgres advisory lock (same primitive Phase 1 already uses for per-draft scheduling).

**Why this is downgraded vs. the old roadmap:** Phase 1 already removed the per-second hot loop. The remaining motivation is purely "I want to deploy web independently of timer code." That's nice but not load-bearing.

---

### Phase 5 — Zero-downtime web deploys via Traefik + sticky sessions (2–3 days)

**The problem:** even after Phase 1, restarting the web container still drops sockets.

**The change:**
- Run **2 web containers** behind Traefik (prerequisites from Phase 3 must be done first).
- Enable **sticky sessions** in Traefik via labels: `traefik.http.services.brackt.loadbalancer.sticky.cookie=true`. Socket.IO needs this — clients must return to the same instance until you add a pub/sub adapter.
- Rolling deploy: take instance A out of Traefik's pool → wait for connections to drain (or hard-cut, since reconnect works) → deploy A → put back in pool → repeat for B.
- This works **without Redis** because each draft room's clients are sticky to one instance.
- For broadcasts that need to reach *both* instances (e.g., timer scheduled on instance A, but admin action on instance B): use Postgres `LISTEN/NOTIFY` (⚠️ verify pgbouncer mode first) or a pending-broadcasts table.

**Why this works at your scale:** with <20 concurrent drafts, a single web instance has zero capacity issues. The second instance is purely for deploy rotation.

**Files:** `.production-info/docker-compose-production.yml` (replicas + sticky labels), deploy script.

---

### Phase 6 — Defer: Socket.IO Redis adapter (only when needed)

**When to revisit:** if you ever want a single draft's clients to span multiple web instances (e.g., 50+ concurrent connections per draft, or geo-distributed users). You're nowhere near that.

**If/when you do:** add Redis, install `@socket.io/redis-adapter`, drop sticky sessions. ~1-day change if Phases 1–5 are clean.

---

### ~~Phase 7 — Go tick rewrite~~ (removed)

Phase 1 (deadline-based timer) eliminates the hot loop the Go rewrite was meant to optimize. The remaining scheduler is `setTimeout` and a Postgres query — Node handles this for free.

If you want to learn Go, pick a greenfield side service (a webhook receiver for scoring feeds, for instance). Don't put it on the critical path of live drafts.

---

## Sequencing Summary

| # | Phase | Effort | Unblocks |
|---|---|---|---|
| 0 | Instrumentation | 1–2d | Every decision below |
| 1 | **Deadline-based draft timer** | 3–5d | Eliminates the 1s tick; makes everything else easier |
| 2 | Snapshots → external HTTP cron | 1d | Removes the last `setInterval`; clean restart semantics |
| 3 | Prod hardening (no `container_name`, healthcheck, pinned tag) | 1–2d | Required for any multi-replica work |
| 4 | Split tick into own process | ~1wk | Optional after Phase 1; deploy isolation only |
| 5 | 2 web instances + Traefik sticky sessions | 2–3d | Zero-downtime deploys (the actual goal) |
| 6 | Redis + socket.io adapter | *deferred* | Cross-instance fan-out (not needed at current scale) |

**Total to "I can deploy without dropping connections": ~2 weeks of focused work** (Phases 0, 1, 2, 3, 5; Phase 4 optional).

---

## Open questions

**Resolved (Vultr managed Postgres):**

- **Pgbouncer modes:** Vultr ships pgbouncer built-in and supports all three modes (session, transaction, statement). You can create **multiple pools with different modes per cluster** via the Vultr panel — so the "pgbouncer incompatibility" concern for pg-boss / `LISTEN/NOTIFY` is *not* a hard blocker; it's "use a session-mode pool for those workloads, transaction-mode for the main app." *Action when relevant:* confirm what pool mode `PGBOUNCER=true` currently routes through (transaction is the typical default) and add a second session-mode pool the day you adopt pg-boss or `LISTEN/NOTIFY`.
- **`pg_cron`:** supported on Vultr (`CREATE EXTENSION IF NOT EXISTS pg_cron;`). Doesn't change the Phase 2 recommendation (external HTTP cron remains simpler for one job) but stays in the toolbox.

**Still open:**

- **Where do logs/metrics go?** Pick one in Phase 0 and commit.

---

## Critical files

- `server.ts` — main entrypoint; stops calling `startSnapshotSystem` (Phase 2); timer system becomes deadline-scheduler (Phase 1).
- `server/timer.ts` — rewrite to deadline-based scheduler (Phase 1); add tick drift metric *before* the rewrite to baseline (Phase 0).
- `server/snapshots.ts` — delete or repurpose; logic moves behind an admin endpoint (Phase 2).
- `server/socket.ts` — emit `{ deadline, serverNow }` events instead of per-second `timer-update` (Phase 1).
- `app/hooks/useDraftSocketEvents.ts` and draft room route — client-side countdown interpolation from deadline (Phase 1).
- `app/database/schema.ts` — add `pickDeadlineAt` to `seasons` (Phase 1).
- `.production-info/docker-compose-production.yml` — remove `container_name`, add healthcheck, pin tag, then add replicas + sticky labels (Phases 3, 5).
- New `app/routes/admin/jobs.run-daily-snapshots.tsx` — HTTP-triggered snapshot job (Phase 2).
- New `app/routes/healthz.tsx` — Traefik healthcheck target (Phase 3).

## Verification

- **Phase 0:** confirm metrics emit (grep logs for tick drift, auto-pick latency); leave running for a week before Phase 1 ships.
- **Phase 1:** run side-by-side mode in a staging draft — deadline-based writes shadow columns, setInterval still drives picks; confirm zero drift, then flip. Then: restart the process mid-draft, confirm timer keeps the right deadline and auto-pick fires at the original time.
- **Phase 2:** kill the web process mid-day; confirm the external cron still triggers tomorrow and the endpoint runs successfully.
- **Phase 3:** `curl /healthz` returns 200 only after DB is reachable; deploy with a pinned tag and confirm you can roll back by changing one line.
- **Phase 5:** rolling-deploy both web instances during a staging draft; users see at most a single brief reconnect (or none, with proper drain).