brackt/docs/infrastructure-roadmap.md

Infrastructure Roadmap for brackt.com

Context

brackt.com today runs as a single Node.js process that bundles three concerns:

1. The Express + React Router 7 SSR web server (server.ts)
2. A 1-second setInterval tick loop driving draft timers and auto-picks (server/timer.ts)
3. A Socket.IO server with in-memory room/connection state (server/socket.ts)

Plus a 24h snapshot interval (server/snapshots.ts). There is no job queue, no scheduler beyond setInterval, and no pub/sub layer. Deploys drop all socket connections; a crash takes down drafts, web, and the tick simultaneously.

Based on the discovery questions:

• No HA fire. Drafts are infrequent enough that a 30-min outage isn't catastrophic yet.
• Zero-downtime deploys are the immediate pain. Connections drop on every release.
• Low concurrency today (<5 live drafts, <20 in 6mo) — capacity is not the constraint.
• Avoid Redis unless clearly justified; prefer Postgres-backed.
• Go rewrite is speculative. No measurements show Node is the bottleneck.

The roadmap below sequences work by risk-adjusted value: cheap, reversible wins first; expensive rewrites only when data demands them.

---

Current production topology

What's actually deployed today (per .production-info/docker-compose-production.yml):

• Single brackt container running on Vultr, image pulled from sjc.vultrcr.com/chrisparsons/brackt:latest.
• One-shot migrate container runs node /app/scripts/migrate.mjs before brackt starts (depends_on: condition: service_completed_successfully).
• Traefik is already fronting the app — TLS via Let's Encrypt, brackt.com apex + www → apex redirect, port 3000.
• External Postgres (not in this compose). PGBOUNCER=true is set in env — the app connects through Vultr's built-in pgbouncer. Vultr supports per-pool mode selection (session/transaction/statement), so worst case is "add a second pool" rather than re-architecting.
• No observability stack. No Prometheus, Loki, Grafana, no log shipper. Logs live in docker logs brackt.
• No Redis, no separate worker, no tick container. Everything is one process.
• restart_policy: condition: on-failure, max_attempts: 5 — after 5 crashes the container stays down with no alerting.
• Hidden blockers for any multi-replica future: container_name: brackt is hardcoded (Docker won't allow two containers with the same name), no healthcheck on the brackt service, and the image tag is :latest (rollbacks are painful and you can't tell which replica is on which version mid-deploy).

These facts shape the phases below — particularly the deadline-based timer recommendation that sidesteps several of these constraints entirely.

---

Roadmap

Phase 0 — Instrumentation (1–2 days) ⭐ do this first

You cannot make further sequencing decisions without data. Almost everything below ("is the tick slow?", "do we need multi-instance?", "is the timer refactor working?") becomes guesswork without basic metrics.

Add:

• Tick drift measurement in server/timer.ts — log when actual interval > 1100ms.
• Auto-pick latency: time from timeRemaining <= 0 → executeAutoPick() resolved.
• Concurrent socket connections + active draft rooms gauge.
• Basic HTTP request duration histogram (any lightweight middleware).

Where to look at the data — pick one, don't leave this abstract:

Option	Effort	What you get
Structured JSON logs to stdout, view via docker logs brackt | jq	~2 hours	Good enough for low-volume signals (tick drift, auto-pick latency). Start here.
Grafana + Loki + Promtail as sidecars in the compose	~1 day	Real dashboards, log retention, query history
Hosted (Axiom / Better Stack / Grafana Cloud free tier)	~half day	No infra to run; ship logs over the wire

Recommendation: start with structured JSON + jq for a week, escalate only if you find yourself wanting historical queries you can't get from logs.

Why first: every decision below is currently a guess. A week of metrics turns guesses into decisions, and gives the Phase 1 timer refactor a baseline to compare against.

---

Phase 1 — Deadline-based draft timer (3–5 days) ⭐ biggest win

The problem: the 1-second setInterval in server/timer.ts is the cause of nearly every downstream problem in this roadmap. Per active draft, every second it issues an UPDATE on draftTimers, several SELECTs, and a timer-update socket broadcast. It also makes the process un-killable without dropping drafts, makes multi-instance hard, and is the only thing in this stack that looks like it might need Go.

The insight: the timer state is deterministic from a deadline. If you store pickDeadlineAt (absolute UTC) instead of timeRemaining (seconds), the remaining time at any moment is just pickDeadlineAt − now(). The server doesn't need to tick — it only needs to wake up at the deadline to run the auto-pick.

The change:

Schema (one new column total):

seasons
  + pickDeadlineAt timestamptz NULL   -- the active deadline for the current pick;
                                      -- NULL while paused, pre-draft, or completed

Why on seasons and not draftTimers: only one team's deadline matters at a time (the current picker's). One row per draft matches the semantics and gives a trivial bootstrap query (SELECT id FROM seasons WHERE status='draft' AND pickDeadlineAt IS NOT NULL). The existing draftTimers.timeRemaining stays where it is — still needed for chess-clock bank tracking and as the "saved remaining seconds while paused" storage.

Server behavior — funnel every deadline write through a single helper:

async function setNextPickDeadline(tx, { seasonId, teamId, budgetSeconds }) {
  // 1. UPDATE seasons SET pickDeadlineAt = now() + budgetSeconds
  // 2. UPDATE draftTimers SET timeRemaining = budgetSeconds (chess-clock: add increment)
  // 3. (re)schedule in-memory setTimeout for this seasonId
  // 4. emit { deadline, serverNow } on draft-${seasonId}
}

Callsites (audit surface):

• Draft auto-start (app/services/draft-autostart.ts) — first pick's deadline.
• Manual pick (app/routes/api/draft.make-pick.ts) — next pick's deadline (currently writes timeRemaining only).
• Auto-pick (server/timer.ts rewrite) — next pick's deadline after auto-picking.
• Admin time-bank adjust (app/routes/api/draft.adjust-time-bank.ts) — recompute pickDeadlineAt to match new timeRemaining.
• Resume from pause — same as auto-start but for the current picker.

Other behaviors:

• Process restart: on boot, query active drafts where pickDeadlineAt IS NOT NULL and re-schedule a setTimeout for each. Already-expired deadlines auto-pick immediately.
• Pause: save remaining into existing draftTimers.timeRemaining (= pickDeadlineAt − now()), null out seasons.pickDeadlineAt, clear the in-memory timeout. No new "paused remaining" column needed — timeRemaining already serves this purpose.
• Overnight pause: when scheduling, if the deadline falls inside the pause window, schedule for resumeAtUTC instead and emit a frozen flag.
• while_on autodraft: unchanged semantics — fire immediately on pick transition, capped at totalTeams iterations. The logic moves from the tick to the pick-completion handler (inside setNextPickDeadline or just before it).

Client behavior:

• Receive { deadline, serverNow }; compute clock offset once.
• Render countdown locally from deadline − clientNow() + offset. Already has to do this between 1Hz emits today, so the interpolation code likely exists.

Wins vs today:

	1s setInterval (today)	Deadline-based
DB writes per active draft per pick	~30–120	1
Socket emits per active draft per pick	~30–120	1 (deadline) + 1 (pick made)
Server wake-ups per second	1 always	0 (sleeps until deadline)
Survives process restart	dies	one-line bootstrap query
Cross-process / multi-instance	hard (singleton tick required)	trivial (whichever instance handles the pick schedules the next deadline)

Critical constraints:

• Single scheduler assumption. Phase 1 runs in one process. Multi-instance leader-election (per-seasonId advisory lock + claim metadata) is deferred to Phase 5. Document this assumption explicitly in server/timer.ts.
• Clock skew. Send serverNow with each deadline event so clients can correct.
• Reconnect path. The loader must return pickDeadlineAt (and serverNow) so clients re-initializing after a reconnect render the correct countdown without waiting for a socket event.
• Migration safety. Run both systems in parallel for one draft (deadline-based writes the new column; old setInterval still drives picks). Compare. Flip when confident.

Why this matters more than splitting the tick into its own process: it dissolves the problem rather than relocating it. Once the tick is deadline-based and idle 99% of the time, splitting it into its own container becomes a deploy-isolation play, not a correctness one. And the Phase 5 Go rewrite goes off the table permanently — there is no hot loop left to optimize.

Files: server/timer.ts (rewrite as a scheduler module), app/database/schema.ts + new migration (single column on seasons), app/models/draft-timer.ts (new setNextPickDeadline helper — see implementation guide), app/services/draft-autostart.ts (call helper on draft start), app/routes/api/draft.make-pick.ts (call helper on manual pick), app/routes/api/draft.adjust-time-bank.ts (recompute deadline on adjust), app/hooks/useDraftSocketEvents.ts (sole timer-update consumer — switch to deadline event), draft room loader (app/routes/leagues/$leagueId.draft.$seasonId.tsx), socket event types.

See docs/agents/deadline-based-timer.md for the implementation guide — schema, helper signature, callsite-by-callsite migration, test plan.

Gotchas to handle (load-bearing for correctness):

1. Stale in-memory setTimeout after out-of-band DB writes. Anything that mutates pickDeadlineAt outside the scheduler (admin extend, retro edit) leaves the captured closure stale. Mitigation: the auto-pick callback must re-read the row and verify pickDeadlineAt still equals the expected value before firing; if changed, reschedule and return. This single guard also handles wall-clock vs. monotonic drift (NTP jumps, VM resume): the re-check catches "deadline already passed" and "deadline moved" with the same code path.
2. Pause-during-flight race. setTimeout fires → enters auto-pick logic → admin pauses concurrently. The re-read in (1) must also check draftPaused before proceeding. Contract: in-flight picks complete; future scheduled picks don't fire after pause.
3. Idempotency on restart. Bootstrap query (pickDeadlineAt IS NOT NULL) fires all expired deadlines on boot. Idempotency is already guaranteed by executeAutoPick (app/models/draft-utils.ts:499–512 pre-check + :629–650 ON CONFLICT DO NOTHING on (seasonId, pickNumber)). Add an explicit test: "auto-pick is idempotent across process restart."
4. Transaction boundary. Pick row + next pickDeadlineAt write must share one transaction. Otherwise a crash between them leaves the draft "completed pick / no deadline" — the bootstrap query won't catch it.

Lower-severity edge cases (worth knowing, not blocking):

• setTimeout 24.8-day overflow: irrelevant for per-pick deadlines; don't reuse this primitive for season.draftDateTime auto-start — keep that on a low-frequency (30–60s) setInterval poll.
• while_on cascade moves from the tick to the pick-completion handler: check next team's autodraft mode before scheduling next deadline; fire immediately if while_on, capped at totalTeams iterations.
• Client tab backgrounding is better than today: browsers throttle timers, but visibilitychange → visible triggers a recompute from the current deadline, so countdowns snap to truth instead of waiting for the next server emit.
• Clock skew: server includes serverNow with every deadline event; client computes a one-time offset.
• Migration of in-flight drafts: two-stage deploy — first ship shadow writes alongside the existing tick (observe a day), then flip the authoritative path and drop the interval.
• Multi-instance scheduling (Phase 5 inheritance): defer the leader-election design; document the "single scheduler" assumption explicitly. The bootstrap query recovers a crashed scheduler on restart, which is sufficient for one instance.
• Observability swap: the Phase 0 "tick drift" metric becomes meaningless after Phase 1. Replace with deadline-fire accuracy = actualFireTime − scheduledFireTime. Same intent.

---

Phase 2 — Scheduled work: snapshots off setInterval (1 day)

The problem: server/snapshots.ts uses a 24h setInterval that dies with the process, has no retry, and no visibility. As you add standings sync, scoring imports, EV snapshots, etc., this gets worse.

The realistic options (the original roadmap assumed pg-boss; given the production constraints, simpler choices win here):

Option	Pros	Cons	Verdict
External HTTP cron (healthchecks.io free / GitHub Actions schedule: / cron-job.org) hitting POST /admin/jobs/run-daily-snapshots	Zero new infra; dead-man-switch alerting built-in; sidesteps pgbouncer entirely	External dependency; needs auth on the endpoint	⭐ start here
node-cron in a small worker container with Postgres advisory lock for singleton	Self-contained; trivial; no 3rd-party	Build basic retry/durability yourself; another container	Good middle ground
pg-boss	Real job system, retries, audit table	Uses LISTEN/NOTIFY and prepared statements — incompatible with pgbouncer transaction pooling. On Vultr, this is solvable: create a second session-mode pool just for pg-boss and pass its DSN to that workload only. No built-in UI.	Defer until you have 5+ scheduled jobs
pg_cron extension	Runs inside Postgres; rock solid; no scheduler process. Supported on Vultr (CREATE EXTENSION IF NOT EXISTS pg_cron;).	Logic lives in SQL; for app-level work you'd call out via pg_net → POST /admin/jobs/snapshots, which is just external HTTP cron with extra steps	Skip for one job; reconsider if you accumulate many

Recommendation: external HTTP cron. The "job" becomes an authenticated POST /admin/jobs/run-daily-snapshots route that calls the existing createSnapshotsForAllSeasons(). Healthchecks.io's free tier gives you "you didn't run today" alerts for free.

Why now (not later): it's a 1-hour change once you have an admin endpoint, and it removes the last setInterval from server.ts — which makes Phase 3 (process restart safety) trivially correct rather than "hopefully correct."

Files: new admin route app/routes/admin/jobs.run-daily-snapshots.tsx (POST handler with shared-secret auth), remove startSnapshotSystem() call from server.ts, delete or repurpose server/snapshots.ts.

---

Phase 3 — Production hardening prerequisites (1–2 days)

Before any multi-replica / zero-downtime work, the production compose needs three small fixes. These are cheap and worth doing regardless of what comes next.

The changes:

1. Remove hardcoded container_name: brackt. Docker rejects duplicate names — this alone blocks scaling to 2 replicas. Use the service name + replica index (Compose handles this automatically when container_name is omitted).
2. Add a healthcheck to the brackt service (e.g., GET /healthz returns 200 once the server is listening and DB is reachable). Without this, Traefik will route to a half-started container during rolling deploys.
3. Pin the image tag. Move off :latest to a Git SHA or version. Rollbacks become a one-line edit; you can tell which replica is running which version during a rolling deploy.

Files: .production-info/docker-compose-production.yml, new app/routes/healthz.tsx, deploy script (whatever sets the tag).

---

Phase 4 — Optional: split tick into its own process (1 week, no longer urgent)

Status after Phase 1: the "tick" is now an idle process that wakes up at deadlines. It can be co-located with web indefinitely without correctness or performance issues.

When to revisit: if you want pure deploy isolation — restart the web tier without touching the timer scheduler at all. This is a nice-to-have, not a blocker.

The change (if/when):

• Extract timer scheduling into server/tick.ts standalone entrypoint.
• Run as a second container in compose.
• Tick still emits via Socket.IO — either via Postgres LISTEN/NOTIFY to the web tier (⚠️ verify pgbouncer mode supports session-scoped listeners) or via a pending-broadcasts table the web tier polls. With Phase 1 done, the broadcast volume is tiny — polling is fine.
• Singleton enforced via Postgres advisory lock (same primitive Phase 1 already uses for per-draft scheduling).

Why this is downgraded vs. the old roadmap: Phase 1 already removed the per-second hot loop. The remaining motivation is purely "I want to deploy web independently of timer code." That's nice but not load-bearing.

---

Phase 5 — Zero-downtime web deploys via Traefik + sticky sessions (2–3 days)

The problem: even after Phase 1, restarting the web container still drops sockets.

The change:

• Run 2 web containers behind Traefik (prerequisites from Phase 3 must be done first).
• Enable sticky sessions in Traefik via labels: traefik.http.services.brackt.loadbalancer.sticky.cookie=true. Socket.IO needs this — clients must return to the same instance until you add a pub/sub adapter.
• Rolling deploy: take instance A out of Traefik's pool → wait for connections to drain (or hard-cut, since reconnect works) → deploy A → put back in pool → repeat for B.
• This works without Redis because each draft room's clients are sticky to one instance.
• For broadcasts that need to reach both instances (e.g., timer scheduled on instance A, but admin action on instance B): use Postgres LISTEN/NOTIFY (⚠️ verify pgbouncer mode first) or a pending-broadcasts table.

Why this works at your scale: with <20 concurrent drafts, a single web instance has zero capacity issues. The second instance is purely for deploy rotation.

Files: .production-info/docker-compose-production.yml (replicas + sticky labels), deploy script.

---

Phase 6 — Defer: Socket.IO Redis adapter (only when needed)

When to revisit: if you ever want a single draft's clients to span multiple web instances (e.g., 50+ concurrent connections per draft, or geo-distributed users). You're nowhere near that.

If/when you do: add Redis, install @socket.io/redis-adapter, drop sticky sessions. ~1-day change if Phases 1–5 are clean.

---

Phase 7 — Go tick rewrite (removed)

Phase 1 (deadline-based timer) eliminates the hot loop the Go rewrite was meant to optimize. The remaining scheduler is setTimeout and a Postgres query — Node handles this for free.

If you want to learn Go, pick a greenfield side service (a webhook receiver for scoring feeds, for instance). Don't put it on the critical path of live drafts.

---

Sequencing Summary

#	Phase	Effort	Unblocks
0	Instrumentation	1–2d	Every decision below
1	Deadline-based draft timer	3–5d	Eliminates the 1s tick; makes everything else easier
2	Snapshots → external HTTP cron	1d	Removes the last setInterval; clean restart semantics
3	Prod hardening (no container_name, healthcheck, pinned tag)	1–2d	Required for any multi-replica work
4	Split tick into own process	~1wk	Optional after Phase 1; deploy isolation only
5	2 web instances + Traefik sticky sessions	2–3d	Zero-downtime deploys (the actual goal)
6	Redis + socket.io adapter	deferred	Cross-instance fan-out (not needed at current scale)

Total to "I can deploy without dropping connections": ~2 weeks of focused work (Phases 0, 1, 2, 3, 5; Phase 4 optional).

---

Open questions

Resolved (Vultr managed Postgres):

• Pgbouncer modes: Vultr ships pgbouncer built-in and supports all three modes (session, transaction, statement). You can create multiple pools with different modes per cluster via the Vultr panel — so the "pgbouncer incompatibility" concern for pg-boss / LISTEN/NOTIFY is not a hard blocker; it's "use a session-mode pool for those workloads, transaction-mode for the main app." Action when relevant: confirm what pool mode PGBOUNCER=true currently routes through (transaction is the typical default) and add a second session-mode pool the day you adopt pg-boss or LISTEN/NOTIFY.
• pg_cron: supported on Vultr (CREATE EXTENSION IF NOT EXISTS pg_cron;). Doesn't change the Phase 2 recommendation (external HTTP cron remains simpler for one job) but stays in the toolbox.

Still open:

• Where do logs/metrics go? Pick one in Phase 0 and commit.

---

Critical files

• server.ts — main entrypoint; stops calling startSnapshotSystem (Phase 2); timer system becomes deadline-scheduler (Phase 1).
• server/timer.ts — rewrite to deadline-based scheduler (Phase 1); add tick drift metric before the rewrite to baseline (Phase 0).
• server/snapshots.ts — delete or repurpose; logic moves behind an admin endpoint (Phase 2).
• server/socket.ts — emit { deadline, serverNow } events instead of per-second timer-update (Phase 1).
• app/hooks/useDraftSocketEvents.ts and draft room route — client-side countdown interpolation from deadline (Phase 1).
• app/database/schema.ts — add pickDeadlineAt to seasons (Phase 1).
• .production-info/docker-compose-production.yml — remove container_name, add healthcheck, pin tag, then add replicas + sticky labels (Phases 3, 5).
• New app/routes/admin/jobs.run-daily-snapshots.tsx — HTTP-triggered snapshot job (Phase 2).
• New app/routes/healthz.tsx — Traefik healthcheck target (Phase 3).

Verification

• Phase 0: confirm metrics emit (grep logs for tick drift, auto-pick latency); leave running for a week before Phase 1 ships.
• Phase 1: run side-by-side mode in a staging draft — deadline-based writes shadow columns, setInterval still drives picks; confirm zero drift, then flip. Then: restart the process mid-draft, confirm timer keeps the right deadline and auto-pick fires at the original time.
• Phase 2: kill the web process mid-day; confirm the external cron still triggers tomorrow and the endpoint runs successfully.
• Phase 3: curl /healthz returns 200 only after DB is reachable; deploy with a pinned tag and confirm you can roll back by changing one line.
• Phase 5: rolling-deploy both web instances during a staging draft; users see at most a single brief reconnect (or none, with proper drain).