Update infrastructure roadmap: mark Phase 1 + Phase A complete, document Phase B/C

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Chris Parsons 2026-06-08 00:45:03 -07:00
parent 273742f02e
commit 968aa72992

View file

@ -2,280 +2,191 @@
## Context
brackt.com today runs as a **single Node.js process** that bundles three concerns:
brackt.com runs as a **single Node.js process** on Vultr, fronted by Traefik, backed by managed Postgres via pgbouncer.
1. The Express + React Router 7 SSR web server (`server.ts`)
2. A 1-second `setInterval` tick loop driving draft timers and auto-picks (`server/timer.ts`)
3. A Socket.IO server with **in-memory** room/connection state (`server/socket.ts`)
Plus a 24h snapshot interval (`server/snapshots.ts`). There is no job queue, no scheduler beyond `setInterval`, and no pub/sub layer. Deploys drop all socket connections; a crash takes down drafts, web, and the tick simultaneously.
Based on the discovery questions:
Based on the original discovery:
- **No HA fire.** Drafts are infrequent enough that a 30-min outage isn't catastrophic *yet*.
- **Zero-downtime deploys are the immediate pain.** Connections drop on every release.
- **Low concurrency today** (<5 live drafts, <20 in 6mo) capacity is not the constraint.
- **Avoid Redis** unless clearly justified; prefer Postgres-backed.
- **Go rewrite is speculative.** No measurements show Node is the bottleneck.
The roadmap below sequences work by **risk-adjusted value**: cheap, reversible wins first; expensive rewrites only when data demands them.
The roadmap sequences work by **risk-adjusted value**: cheap, reversible wins first.
---
## Current production topology
What's actually deployed today (per `.production-info/docker-compose-production.yml`):
As of Phase A (merged 2026-06-08):
- **Single `brackt` container** running on Vultr, image pulled from `sjc.vultrcr.com/chrisparsons/brackt:latest`.
- **One-shot `migrate` container** runs `node /app/scripts/migrate.mjs` before brackt starts (`depends_on: condition: service_completed_successfully`).
- **Traefik** is already fronting the app — TLS via Let's Encrypt, `brackt.com` apex + `www` → apex redirect, port 3000.
- **External Postgres** (not in this compose). `PGBOUNCER=true` is set in env — the app connects through Vultr's built-in pgbouncer. Vultr supports per-pool mode selection (session/transaction/statement), so worst case is "add a second pool" rather than re-architecting.
- **No observability stack.** No Prometheus, Loki, Grafana, no log shipper. Logs live in `docker logs brackt`.
- **No Redis, no separate worker, no tick container.** Everything is one process.
- `restart_policy: condition: on-failure, max_attempts: 5` — after 5 crashes the container stays down with no alerting.
- **Hidden blockers for any multi-replica future:** `container_name: brackt` is hardcoded (Docker won't allow two containers with the same name), no healthcheck on the brackt service, and the image tag is `:latest` (rollbacks are painful and you can't tell which replica is on which version mid-deploy).
These facts shape the phases below — particularly the deadline-based timer recommendation that sidesteps several of these constraints entirely.
- **Single `brackt` container** running on Vultr, image `sjc.vultrcr.com/chrisparsons/brackt:latest`.
- **One-shot `migrate` container** runs migrations before brackt starts.
- **Traefik** fronts the app — TLS via Let's Encrypt, apex + www redirect.
- **External Postgres** via Vultr's built-in pgbouncer (`PGBOUNCER=true`).
- **No in-process `setInterval` for timers or snapshots** — draft timer is deadline-based (`picksExpiresAt`); snapshots/standings/simulations are driven by external Forgejo Actions cron jobs hitting authenticated HTTP endpoints.
- **No observability stack.** Logs live in `docker logs brackt`.
- **Still present:** `container_name: brackt` (blocks replicas), no healthcheck, `:latest` tag (painful rollbacks). These are Phase B.
---
## Roadmap
### Phase 0 — Instrumentation (12 days) ⭐ do this first
### ✅ Phase 1 — Deadline-based draft timer (complete)
You cannot make further sequencing decisions without data. Almost everything below ("is the tick slow?", "do we need multi-instance?", "is the timer refactor working?") becomes guesswork without basic metrics.
The 1-second `setInterval` in `server/timer.ts` has been replaced with per-season `setTimeout` calls keyed to `picksExpiresAt` (a `timestamptz` on `draftTimers`). On process restart, active deadlines are recovered from the DB and rescheduled.
**Add:**
- Tick drift measurement in `server/timer.ts` — log when actual interval > 1100ms.
- Auto-pick latency: time from `timeRemaining <= 0``executeAutoPick()` resolved.
- Concurrent socket connections + active draft rooms gauge.
- Basic HTTP request duration histogram (any lightweight middleware).
**Result:** zero per-second DB writes or socket emits during a draft. The server wakes only at pick deadlines. All callsites updated: `make-pick`, `adjust-time-bank`, draft loader, `useDraftSocketEvents`.
**Where to look at the data — pick one, don't leave this abstract:**
---
| Option | Effort | What you get |
### ✅ Phase A — External HTTP cron jobs (complete, merged 2026-06-08)
Replaced the last in-process `setInterval` (`server/snapshots.ts` 24h loop) with Forgejo Actions scheduled workflows hitting authenticated endpoints.
**New endpoints** (all behind `X-Cron-Secret` header):
| Endpoint | Schedule | What it does |
|---|---|---|
| Structured JSON logs to stdout, view via `docker logs brackt \| jq` | ~2 hours | Good enough for low-volume signals (tick drift, auto-pick latency). Start here. |
| Grafana + Loki + Promtail as sidecars in the compose | ~1 day | Real dashboards, log retention, query history |
| Hosted (Axiom / Better Stack / Grafana Cloud free tier) | ~half day | No infra to run; ship logs over the wire |
| `POST /admin/jobs/run-daily-snapshots` | `5 0 * * *` | Daily fantasy standings snapshots for active/draft seasons |
| `POST /admin/jobs/sync-and-simulate` | `0 */2 * * *` | Syncs standings from external APIs; simulates only when standings changed |
| `GET /healthz` | Traefik / Docker healthcheck | 200 `{ok:true}` when DB reachable, 503 otherwise |
**Recommendation:** start with structured JSON + `jq` for a week, escalate only if you find yourself wanting historical queries you can't get from logs.
**Schema additions** (migration 0118): `standingsLastChangedAt`, `lastSimulatedAt` on `sportsSeasons`.
**Why first:** every decision below is currently a guess. A week of metrics turns guesses into decisions, *and* gives the Phase 1 timer refactor a baseline to compare against.
**Change detection:** `syncStandings()` compares `gamesPlayed` + `leagueRank` against current DB rows before upsert; sets `standingsLastChangedAt` only on real change. Simulation skipped if nothing changed.
**Setup required on production:** add `CRON_SECRET` to Forgejo repo secrets and to the server `.env`.
---
### Phase 1 — Deadline-based draft timer (35 days) ⭐ biggest win
### 🔜 Phase B — Production hardening + deploy automation (next, ~1 day)
**The problem:** the 1-second `setInterval` in `server/timer.ts` is the *cause* of nearly every downstream problem in this roadmap. Per active draft, every second it issues an UPDATE on `draftTimers`, several SELECTs, and a `timer-update` socket broadcast. It also makes the process un-killable without dropping drafts, makes multi-instance hard, and is the only thing in this stack that looks like it might need Go.
Three blockers for multi-replica work, plus automating compose file delivery.
**The insight:** the timer state is *deterministic from a deadline*. If you store `pickDeadlineAt` (absolute UTC) instead of `timeRemaining` (seconds), the remaining time at any moment is just `pickDeadlineAt now()`. The server doesn't need to tick — it only needs to wake up *at the deadline* to run the auto-pick.
#### 1. Automate compose file delivery via deploy workflow
**The change:**
Currently `.production-info/docker-compose-production.yml` is a manual reference — the server's `~/brackt/docker-compose.yml` drifts silently. Fix: add an `appleboy/scp-action` step to the deploy job that copies the repo's compose file to the server on every deploy.
Schema (one new column total):
```
seasons
+ pickDeadlineAt timestamptz NULL -- the active deadline for the current pick;
-- NULL while paused, pre-draft, or completed
```
Why on `seasons` and not `draftTimers`: only one team's deadline matters at a time (the current picker's). One row per draft matches the semantics and gives a trivial bootstrap query (`SELECT id FROM seasons WHERE status='draft' AND pickDeadlineAt IS NOT NULL`). The existing `draftTimers.timeRemaining` stays where it is — still needed for chess-clock bank tracking and as the "saved remaining seconds while paused" storage.
**Deploy workflow changes:**
- Add checkout step to the deploy job (needed to access the file)
- Add SCP step: copy `.production-info/docker-compose-production.yml``~/brackt/docker-compose.yml`
- Build with both `:latest` and `:{sha}` tags; pass `IMAGE_TAG=${{ github.sha }}` to the SSH step
- SSH step exports `IMAGE_TAG` before running `docker compose up`
Server behavior — funnel **every** deadline write through a single helper:
```ts
async function setNextPickDeadline(tx, { seasonId, teamId, budgetSeconds }) {
// 1. UPDATE seasons SET pickDeadlineAt = now() + budgetSeconds
// 2. UPDATE draftTimers SET timeRemaining = budgetSeconds (chess-clock: add increment)
// 3. (re)schedule in-memory setTimeout for this seasonId
// 4. emit { deadline, serverNow } on draft-${seasonId}
}
```
Callsites (audit surface):
- **Draft auto-start** (`app/services/draft-autostart.ts`) — first pick's deadline.
- **Manual pick** (`app/routes/api/draft.make-pick.ts`) — next pick's deadline (currently writes `timeRemaining` only).
- **Auto-pick** (`server/timer.ts` rewrite) — next pick's deadline after auto-picking.
- **Admin time-bank adjust** (`app/routes/api/draft.adjust-time-bank.ts`) — recompute `pickDeadlineAt` to match new `timeRemaining`.
- **Resume from pause** — same as auto-start but for the current picker.
#### 2. Move secrets to a server-side `.env` file
Other behaviors:
- **Process restart:** on boot, query active drafts where `pickDeadlineAt IS NOT NULL` and re-schedule a `setTimeout` for each. Already-expired deadlines auto-pick immediately.
- **Pause:** save remaining into existing `draftTimers.timeRemaining` (`= pickDeadlineAt now()`), null out `seasons.pickDeadlineAt`, clear the in-memory timeout. **No new "paused remaining" column needed**`timeRemaining` already serves this purpose.
- **Overnight pause:** when scheduling, if the deadline falls inside the pause window, schedule for `resumeAtUTC` instead and emit a `frozen` flag.
- **`while_on` autodraft:** unchanged semantics — fire immediately on pick transition, capped at `totalTeams` iterations. The logic moves from the tick to the pick-completion handler (inside `setNextPickDeadline` or just before it).
Currently secrets reach the container via unknown means (no `env_file:` in compose, no `environment:` entries for DB credentials). Make this explicit: add `env_file: .env` to the brackt service; document all required vars in `.production-info/.env.production.example`.
Client behavior:
- Receive `{ deadline, serverNow }`; compute clock offset once.
- Render countdown locally from `deadline clientNow() + offset`. Already has to do this between 1Hz emits today, so the interpolation code likely exists.
Non-secret config (NODE_ENV, PGBOUNCER, APP_URL, etc.) stays inline in the compose file. Secrets (DATABASE_URL, BETTER_AUTH_SECRET, CRON_SECRET, RESEND_API_KEY, Cloudinary, Discord, Google, Sentry, Turnstile) go in the server `.env`.
**Wins vs today:**
#### 3. Compose file fixes
| | 1s setInterval (today) | Deadline-based |
|---|---|---|
| DB writes per active draft per pick | ~30120 | 1 |
| Socket emits per active draft per pick | ~30120 | 1 (deadline) + 1 (pick made) |
| Server wake-ups per second | 1 always | 0 (sleeps until deadline) |
| Survives process restart | dies | one-line bootstrap query |
| Cross-process / multi-instance | hard (singleton tick required) | trivial (whichever instance handles the pick schedules the next deadline) |
- **Remove `container_name: brackt`** — Docker rejects duplicate names; this alone blocks `replicas: 2`.
- **Add healthcheck** using `/healthz` (already shipped in Phase A):
```yaml
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3000/healthz"]
interval: 30s
timeout: 5s
retries: 3
start_period: 10s
```
- **Pin image tag** to `${CONTAINER_REGISTRY}/brackt:${IMAGE_TAG:-latest}` — rollback becomes `IMAGE_TAG=<sha> docker compose up`.
**Critical constraints:**
- **Single scheduler assumption.** Phase 1 runs in one process. Multi-instance leader-election (per-`seasonId` advisory lock + claim metadata) is deferred to Phase 5. Document this assumption explicitly in `server/timer.ts`.
- **Clock skew.** Send `serverNow` with each deadline event so clients can correct.
- **Reconnect path.** The loader must return `pickDeadlineAt` (and `serverNow`) so clients re-initializing after a reconnect render the correct countdown without waiting for a socket event.
- **Migration safety.** Run both systems in parallel for one draft (deadline-based writes the new column; old setInterval still drives picks). Compare. Flip when confident.
**Files:**
- `.production-info/docker-compose-production.yml` — remove container_name, add healthcheck, parameterise image tag, add env_file
- `.production-info/.env.production.example` — template for the server-side secrets file
- `.forgejo/workflows/deploy.yml` — add checkout + SCP step, build dual tags, pass IMAGE_TAG
**Why this matters more than splitting the tick into its own process:** it dissolves the problem rather than relocating it. Once the tick is deadline-based and idle 99% of the time, splitting it into its own container becomes a deploy-isolation play, not a correctness one. And the Phase 5 Go rewrite goes off the table permanently — there is no hot loop left to optimize.
**Files:** `server/timer.ts` (rewrite as a scheduler module), `app/database/schema.ts` + new migration (single column on `seasons`), `app/models/draft-timer.ts` (new `setNextPickDeadline` helper — see implementation guide), `app/services/draft-autostart.ts` (call helper on draft start), `app/routes/api/draft.make-pick.ts` (call helper on manual pick), `app/routes/api/draft.adjust-time-bank.ts` (recompute deadline on adjust), `app/hooks/useDraftSocketEvents.ts` (sole `timer-update` consumer — switch to deadline event), draft room loader (`app/routes/leagues/$leagueId.draft.$seasonId.tsx`), socket event types.
**See [`docs/agents/deadline-based-timer.md`](agents/deadline-based-timer.md) for the implementation guide** — schema, helper signature, callsite-by-callsite migration, test plan.
**Gotchas to handle (load-bearing for correctness):**
1. **Stale in-memory `setTimeout` after out-of-band DB writes.** Anything that mutates `pickDeadlineAt` outside the scheduler (admin extend, retro edit) leaves the captured closure stale. *Mitigation:* the auto-pick callback must re-read the row and verify `pickDeadlineAt` still equals the expected value before firing; if changed, reschedule and return. This single guard also handles wall-clock vs. monotonic drift (NTP jumps, VM resume): the re-check catches "deadline already passed" and "deadline moved" with the same code path.
2. **Pause-during-flight race.** `setTimeout` fires → enters auto-pick logic → admin pauses concurrently. The re-read in (1) must also check `draftPaused` before proceeding. Contract: in-flight picks complete; future scheduled picks don't fire after pause.
3. **Idempotency on restart.** Bootstrap query (`pickDeadlineAt IS NOT NULL`) fires all expired deadlines on boot. Idempotency is **already guaranteed** by `executeAutoPick` (`app/models/draft-utils.ts:499512` pre-check + `:629650` `ON CONFLICT DO NOTHING` on `(seasonId, pickNumber)`). Add an explicit test: "auto-pick is idempotent across process restart."
4. **Transaction boundary.** Pick row + next `pickDeadlineAt` write must share one transaction. Otherwise a crash between them leaves the draft "completed pick / no deadline" — the bootstrap query won't catch it.
**Lower-severity edge cases (worth knowing, not blocking):**
- **`setTimeout` 24.8-day overflow:** irrelevant for per-pick deadlines; *don't* reuse this primitive for `season.draftDateTime` auto-start — keep that on a low-frequency (3060s) `setInterval` poll.
- **`while_on` cascade** moves from the tick to the pick-completion handler: check next team's autodraft mode before scheduling next deadline; fire immediately if `while_on`, capped at `totalTeams` iterations.
- **Client tab backgrounding** is *better* than today: browsers throttle timers, but `visibilitychange → visible` triggers a recompute from the current deadline, so countdowns snap to truth instead of waiting for the next server emit.
- **Clock skew:** server includes `serverNow` with every deadline event; client computes a one-time offset.
- **Migration of in-flight drafts:** two-stage deploy — first ship shadow writes alongside the existing tick (observe a day), then flip the authoritative path and drop the interval.
- **Multi-instance scheduling** (Phase 5 inheritance): defer the leader-election design; document the "single scheduler" assumption explicitly. The bootstrap query recovers a crashed scheduler on restart, which is sufficient for one instance.
- **Observability swap:** the Phase 0 "tick drift" metric becomes meaningless after Phase 1. Replace with **deadline-fire accuracy** = `actualFireTime scheduledFireTime`. Same intent.
**Verification:**
- `curl https://brackt.com/healthz` → 200 `{ok:true}`
- Deploy with a SHA tag; confirm rollback by setting `IMAGE_TAG=<previous-sha>` and rerunning compose
- `docker compose up --scale brackt=2` no longer fails with "name already in use"
---
### Phase 2 — Scheduled work: snapshots off `setInterval` (1 day)
### 🔜 Phase C — Zero-downtime deploys via Traefik sticky sessions (~23 days)
**The problem:** `server/snapshots.ts` uses a 24h `setInterval` that dies with the process, has no retry, and no visibility. As you add standings sync, scoring imports, EV snapshots, etc., this gets worse.
Requires Phase B (no `container_name`, healthcheck, pinned tag).
**The realistic options** (the original roadmap assumed pg-boss; given the production constraints, simpler choices win here):
Run 2 web containers behind Traefik with sticky sessions. Rolling deploy takes one replica out of rotation, upgrades it, puts it back — other replica keeps serving.
| Option | Pros | Cons | Verdict |
**Compose changes:**
```yaml
deploy:
replicas: 2
labels:
- "traefik.http.services.brackt.loadbalancer.sticky.cookie=true"
- "traefik.http.services.brackt.loadbalancer.sticky.cookie.name=brackt_instance"
```
Socket.IO clients must return to the same instance (sticky cookie). No Redis adapter needed at current scale — the second instance is purely for deploy rotation, not capacity.
**Rolling deploy script:**
1. Scale down replica A (remove from Traefik pool)
2. Wait for drain (or 5s hard cut — reconnect works)
3. Pull new image on A, `docker compose up` A
4. Healthcheck passes → A back in pool
5. Repeat for B
**Cross-instance broadcasts** (admin action on instance B needs to reach draft clients on instance A): use a pending-broadcasts table polled every 5s by each instance. With deadline-based timers the volume is tiny. Postgres `LISTEN/NOTIFY` is the upgrade path if polling proves too slow — requires a session-mode pgbouncer pool.
**Files:** `.production-info/docker-compose-production.yml` (replicas + sticky labels), rolling deploy script.
**Verification:** rolling-deploy both instances during a staging draft; clients see at most one brief reconnect.
---
### Phase D — Optional: split tick into its own process (~1 week, not urgent)
After Phase 1, the "tick" is an idle process that wakes only at pick deadlines. It can stay co-located with web indefinitely. Only revisit if you want to deploy web without touching timer code at all.
**If/when:** extract `server/tick.ts` as a standalone entrypoint, run as a second compose service, enforce singleton via Postgres advisory lock.
---
### Phase E — Defer: Socket.IO Redis adapter
Only needed if a single draft's clients need to span multiple web instances (e.g., 50+ concurrent connections per draft, or geo-distributed). Not relevant at current scale.
If/when: add Redis, `@socket.io/redis-adapter`, drop sticky sessions. ~1-day change if Phases BC are clean.
---
## Sequencing summary
| Phase | Status | Effort | Key win |
|---|---|---|---|
| **External HTTP cron** (healthchecks.io free / GitHub Actions `schedule:` / cron-job.org) hitting `POST /admin/jobs/run-daily-snapshots` | Zero new infra; dead-man-switch alerting built-in; sidesteps pgbouncer entirely | External dependency; needs auth on the endpoint | ⭐ start here |
| `node-cron` in a small worker container with Postgres advisory lock for singleton | Self-contained; trivial; no 3rd-party | Build basic retry/durability yourself; another container | Good middle ground |
| `pg-boss` | Real job system, retries, audit table | Uses `LISTEN/NOTIFY` and prepared statements — incompatible with pgbouncer transaction pooling. **On Vultr, this is solvable:** create a second session-mode pool just for pg-boss and pass its DSN to that workload only. No built-in UI. | Defer until you have 5+ scheduled jobs |
| `pg_cron` extension | Runs inside Postgres; rock solid; no scheduler process. **Supported on Vultr** (`CREATE EXTENSION IF NOT EXISTS pg_cron;`). | Logic lives in SQL; for app-level work you'd call out via `pg_net → POST /admin/jobs/snapshots`, which is just external HTTP cron with extra steps | Skip for one job; reconsider if you accumulate many |
| 1 | ✅ Done | — | Deadline-based timer; eliminates 1s tick |
| A | ✅ Done | — | External cron; automated standings sync + sim; /healthz |
| B | 🔜 Next | ~1d | Compose delivered by CI; pinned tags; healthcheck; unblocks C |
| C | Blocked on B | ~23d | Zero-downtime rolling deploys |
| D | Optional | ~1wk | Deploy web without restarting timer process |
| E | Deferred | ~1d | Cross-instance socket fan-out (not needed yet) |
**Recommendation:** external HTTP cron. The "job" becomes an authenticated `POST /admin/jobs/run-daily-snapshots` route that calls the existing `createSnapshotsForAllSeasons()`. Healthchecks.io's free tier gives you "you didn't run today" alerts for free.
**Why now (not later):** it's a 1-hour change once you have an admin endpoint, and it removes the last `setInterval` from `server.ts` — which makes Phase 3 (process restart safety) trivially correct rather than "hopefully correct."
**Files:** new admin route `app/routes/admin/jobs.run-daily-snapshots.tsx` (POST handler with shared-secret auth), remove `startSnapshotSystem()` call from `server.ts`, delete or repurpose `server/snapshots.ts`.
---
### Phase 3 — Production hardening prerequisites (12 days)
Before any multi-replica / zero-downtime work, the production compose needs three small fixes. These are cheap and worth doing regardless of what comes next.
**The changes:**
1. **Remove hardcoded `container_name: brackt`.** Docker rejects duplicate names — this alone blocks scaling to 2 replicas. Use the service name + replica index (Compose handles this automatically when `container_name` is omitted).
2. **Add a healthcheck** to the brackt service (e.g., `GET /healthz` returns 200 once the server is listening and DB is reachable). Without this, Traefik will route to a half-started container during rolling deploys.
3. **Pin the image tag.** Move off `:latest` to a Git SHA or version. Rollbacks become a one-line edit; you can tell which replica is running which version during a rolling deploy.
**Files:** `.production-info/docker-compose-production.yml`, new `app/routes/healthz.tsx`, deploy script (whatever sets the tag).
---
### Phase 4 — Optional: split tick into its own process (1 week, no longer urgent)
**Status after Phase 1:** the "tick" is now an idle process that wakes up at deadlines. It can be co-located with web indefinitely without correctness or performance issues.
**When to revisit:** if you want pure deploy isolation — restart the web tier without touching the timer scheduler at all. This is a nice-to-have, not a blocker.
**The change (if/when):**
- Extract timer scheduling into `server/tick.ts` standalone entrypoint.
- Run as a second container in compose.
- Tick still emits via Socket.IO — either via Postgres `LISTEN/NOTIFY` to the web tier (⚠️ verify pgbouncer mode supports session-scoped listeners) or via a pending-broadcasts table the web tier polls. With Phase 1 done, the broadcast volume is tiny — polling is fine.
- Singleton enforced via Postgres advisory lock (same primitive Phase 1 already uses for per-draft scheduling).
**Why this is downgraded vs. the old roadmap:** Phase 1 already removed the per-second hot loop. The remaining motivation is purely "I want to deploy web independently of timer code." That's nice but not load-bearing.
---
### Phase 5 — Zero-downtime web deploys via Traefik + sticky sessions (23 days)
**The problem:** even after Phase 1, restarting the web container still drops sockets.
**The change:**
- Run **2 web containers** behind Traefik (prerequisites from Phase 3 must be done first).
- Enable **sticky sessions** in Traefik via labels: `traefik.http.services.brackt.loadbalancer.sticky.cookie=true`. Socket.IO needs this — clients must return to the same instance until you add a pub/sub adapter.
- Rolling deploy: take instance A out of Traefik's pool → wait for connections to drain (or hard-cut, since reconnect works) → deploy A → put back in pool → repeat for B.
- This works **without Redis** because each draft room's clients are sticky to one instance.
- For broadcasts that need to reach *both* instances (e.g., timer scheduled on instance A, but admin action on instance B): use Postgres `LISTEN/NOTIFY` (⚠️ verify pgbouncer mode first) or a pending-broadcasts table.
**Why this works at your scale:** with <20 concurrent drafts, a single web instance has zero capacity issues. The second instance is purely for deploy rotation.
**Files:** `.production-info/docker-compose-production.yml` (replicas + sticky labels), deploy script.
---
### Phase 6 — Defer: Socket.IO Redis adapter (only when needed)
**When to revisit:** if you ever want a single draft's clients to span multiple web instances (e.g., 50+ concurrent connections per draft, or geo-distributed users). You're nowhere near that.
**If/when you do:** add Redis, install `@socket.io/redis-adapter`, drop sticky sessions. ~1-day change if Phases 15 are clean.
---
### ~~Phase 7 — Go tick rewrite~~ (removed)
Phase 1 (deadline-based timer) eliminates the hot loop the Go rewrite was meant to optimize. The remaining scheduler is `setTimeout` and a Postgres query — Node handles this for free.
If you want to learn Go, pick a greenfield side service (a webhook receiver for scoring feeds, for instance). Don't put it on the critical path of live drafts.
---
## Sequencing Summary
| # | Phase | Effort | Unblocks |
|---|---|---|---|
| 0 | Instrumentation | 12d | Every decision below |
| 1 | **Deadline-based draft timer** | 35d | Eliminates the 1s tick; makes everything else easier |
| 2 | Snapshots → external HTTP cron | 1d | Removes the last `setInterval`; clean restart semantics |
| 3 | Prod hardening (no `container_name`, healthcheck, pinned tag) | 12d | Required for any multi-replica work |
| 4 | Split tick into own process | ~1wk | Optional after Phase 1; deploy isolation only |
| 5 | 2 web instances + Traefik sticky sessions | 23d | Zero-downtime deploys (the actual goal) |
| 6 | Redis + socket.io adapter | *deferred* | Cross-instance fan-out (not needed at current scale) |
**Total to "I can deploy without dropping connections": ~2 weeks of focused work** (Phases 0, 1, 2, 3, 5; Phase 4 optional).
**Total to zero-downtime deploys: ~34 days of focused work** (Phases B + C).
---
## Open questions
**Resolved (Vultr managed Postgres):**
- **Pgbouncer modes:** Vultr ships pgbouncer built-in and supports all three modes (session, transaction, statement). You can create **multiple pools with different modes per cluster** via the Vultr panel — so the "pgbouncer incompatibility" concern for pg-boss / `LISTEN/NOTIFY` is *not* a hard blocker; it's "use a session-mode pool for those workloads, transaction-mode for the main app." *Action when relevant:* confirm what pool mode `PGBOUNCER=true` currently routes through (transaction is the typical default) and add a second session-mode pool the day you adopt pg-boss or `LISTEN/NOTIFY`.
- **`pg_cron`:** supported on Vultr (`CREATE EXTENSION IF NOT EXISTS pg_cron;`). Doesn't change the Phase 2 recommendation (external HTTP cron remains simpler for one job) but stays in the toolbox.
**Still open:**
- **Where do logs/metrics go?** Pick one in Phase 0 and commit.
- **Where do logs/metrics go?** Still unresolved. Options: structured JSON to stdout + `docker logs | jq` (zero effort), hosted (Better Stack / Axiom free tier, ~half day), or Grafana + Loki sidecars (~1 day). Pick one before adding more observability instrumentation.
- **Pgbouncer pool mode:** confirm current pool is transaction mode. Add a second session-mode pool before adopting `LISTEN/NOTIFY` or pg-boss.
---
## Critical files
- `server.ts` — main entrypoint; stops calling `startSnapshotSystem` (Phase 2); timer system becomes deadline-scheduler (Phase 1).
- `server/timer.ts` — rewrite to deadline-based scheduler (Phase 1); add tick drift metric *before* the rewrite to baseline (Phase 0).
- `server/snapshots.ts` — delete or repurpose; logic moves behind an admin endpoint (Phase 2).
- `server/socket.ts` — emit `{ deadline, serverNow }` events instead of per-second `timer-update` (Phase 1).
- `app/hooks/useDraftSocketEvents.ts` and draft room route — client-side countdown interpolation from deadline (Phase 1).
- `app/database/schema.ts` — add `pickDeadlineAt` to `seasons` (Phase 1).
- `.production-info/docker-compose-production.yml` — remove `container_name`, add healthcheck, pin tag, then add replicas + sticky labels (Phases 3, 5).
- New `app/routes/admin/jobs.run-daily-snapshots.tsx` — HTTP-triggered snapshot job (Phase 2).
- New `app/routes/healthz.tsx` — Traefik healthcheck target (Phase 3).
## Verification
- **Phase 0:** confirm metrics emit (grep logs for tick drift, auto-pick latency); leave running for a week before Phase 1 ships.
- **Phase 1:** run side-by-side mode in a staging draft — deadline-based writes shadow columns, setInterval still drives picks; confirm zero drift, then flip. Then: restart the process mid-draft, confirm timer keeps the right deadline and auto-pick fires at the original time.
- **Phase 2:** kill the web process mid-day; confirm the external cron still triggers tomorrow and the endpoint runs successfully.
- **Phase 3:** `curl /healthz` returns 200 only after DB is reachable; deploy with a pinned tag and confirm you can roll back by changing one line.
- **Phase 5:** rolling-deploy both web instances during a staging draft; users see at most a single brief reconnect (or none, with proper drain).
- `server/timer.ts` — deadline-based scheduler; `picksExpiresAt` on `draftTimers`
- `server/socket.ts` — emits `{ expiresAt, timeRemaining }` on `timer-pick-started`
- `app/routes/healthz.ts` — DB ping; used by Docker healthcheck and Traefik
- `app/routes/admin/jobs.run-daily-snapshots.ts` — daily snapshot cron endpoint
- `app/routes/admin/jobs.sync-and-simulate.ts` — standings sync + conditional sim cron endpoint
- `app/lib/cron-auth.ts``requireCronSecret()` helper for cron endpoints
- `app/services/standings-sync/index.ts``syncStandings()` returns `{ changed: boolean }`
- `.production-info/docker-compose-production.yml` — source of truth for production compose (Phase B: delivered by CI)
- `.production-info/.env.production.example` — template for server-side secrets (Phase B)
- `.forgejo/workflows/deploy.yml` — CI/CD pipeline (Phase B: add SCP + dual image tags)
- `.forgejo/workflows/daily-snapshots.yml` — daily snapshot cron (Forgejo Actions)
- `.forgejo/workflows/sync-and-simulate.yml` — 2h standings sync + sim cron (Forgejo Actions)