diff --git a/docs/infrastructure-roadmap.md b/docs/infrastructure-roadmap.md index 039d594..f58d30d 100644 --- a/docs/infrastructure-roadmap.md +++ b/docs/infrastructure-roadmap.md @@ -2,280 +2,191 @@ ## Context -brackt.com today runs as a **single Node.js process** that bundles three concerns: +brackt.com runs as a **single Node.js process** on Vultr, fronted by Traefik, backed by managed Postgres via pgbouncer. -1. The Express + React Router 7 SSR web server (`server.ts`) -2. A 1-second `setInterval` tick loop driving draft timers and auto-picks (`server/timer.ts`) -3. A Socket.IO server with **in-memory** room/connection state (`server/socket.ts`) - -Plus a 24h snapshot interval (`server/snapshots.ts`). There is no job queue, no scheduler beyond `setInterval`, and no pub/sub layer. Deploys drop all socket connections; a crash takes down drafts, web, and the tick simultaneously. - -Based on the discovery questions: +Based on the original discovery: - **No HA fire.** Drafts are infrequent enough that a 30-min outage isn't catastrophic *yet*. - **Zero-downtime deploys are the immediate pain.** Connections drop on every release. - **Low concurrency today** (<5 live drafts, <20 in 6mo) — capacity is not the constraint. - **Avoid Redis** unless clearly justified; prefer Postgres-backed. -- **Go rewrite is speculative.** No measurements show Node is the bottleneck. -The roadmap below sequences work by **risk-adjusted value**: cheap, reversible wins first; expensive rewrites only when data demands them. +The roadmap sequences work by **risk-adjusted value**: cheap, reversible wins first. --- ## Current production topology -What's actually deployed today (per `.production-info/docker-compose-production.yml`): +As of Phase A (merged 2026-06-08): -- **Single `brackt` container** running on Vultr, image pulled from `sjc.vultrcr.com/chrisparsons/brackt:latest`. -- **One-shot `migrate` container** runs `node /app/scripts/migrate.mjs` before brackt starts (`depends_on: condition: service_completed_successfully`). -- **Traefik** is already fronting the app — TLS via Let's Encrypt, `brackt.com` apex + `www` → apex redirect, port 3000. -- **External Postgres** (not in this compose). `PGBOUNCER=true` is set in env — the app connects through Vultr's built-in pgbouncer. Vultr supports per-pool mode selection (session/transaction/statement), so worst case is "add a second pool" rather than re-architecting. -- **No observability stack.** No Prometheus, Loki, Grafana, no log shipper. Logs live in `docker logs brackt`. -- **No Redis, no separate worker, no tick container.** Everything is one process. -- `restart_policy: condition: on-failure, max_attempts: 5` — after 5 crashes the container stays down with no alerting. -- **Hidden blockers for any multi-replica future:** `container_name: brackt` is hardcoded (Docker won't allow two containers with the same name), no healthcheck on the brackt service, and the image tag is `:latest` (rollbacks are painful and you can't tell which replica is on which version mid-deploy). - -These facts shape the phases below — particularly the deadline-based timer recommendation that sidesteps several of these constraints entirely. +- **Single `brackt` container** running on Vultr, image `sjc.vultrcr.com/chrisparsons/brackt:latest`. +- **One-shot `migrate` container** runs migrations before brackt starts. +- **Traefik** fronts the app — TLS via Let's Encrypt, apex + www redirect. +- **External Postgres** via Vultr's built-in pgbouncer (`PGBOUNCER=true`). +- **No in-process `setInterval` for timers or snapshots** — draft timer is deadline-based (`picksExpiresAt`); snapshots/standings/simulations are driven by external Forgejo Actions cron jobs hitting authenticated HTTP endpoints. +- **No observability stack.** Logs live in `docker logs brackt`. +- **Still present:** `container_name: brackt` (blocks replicas), no healthcheck, `:latest` tag (painful rollbacks). These are Phase B. --- ## Roadmap -### Phase 0 — Instrumentation (1–2 days) ⭐ do this first +### ✅ Phase 1 — Deadline-based draft timer (complete) -You cannot make further sequencing decisions without data. Almost everything below ("is the tick slow?", "do we need multi-instance?", "is the timer refactor working?") becomes guesswork without basic metrics. +The 1-second `setInterval` in `server/timer.ts` has been replaced with per-season `setTimeout` calls keyed to `picksExpiresAt` (a `timestamptz` on `draftTimers`). On process restart, active deadlines are recovered from the DB and rescheduled. -**Add:** -- Tick drift measurement in `server/timer.ts` — log when actual interval > 1100ms. -- Auto-pick latency: time from `timeRemaining <= 0` → `executeAutoPick()` resolved. -- Concurrent socket connections + active draft rooms gauge. -- Basic HTTP request duration histogram (any lightweight middleware). +**Result:** zero per-second DB writes or socket emits during a draft. The server wakes only at pick deadlines. All callsites updated: `make-pick`, `adjust-time-bank`, draft loader, `useDraftSocketEvents`. -**Where to look at the data — pick one, don't leave this abstract:** +--- -| Option | Effort | What you get | +### ✅ Phase A — External HTTP cron jobs (complete, merged 2026-06-08) + +Replaced the last in-process `setInterval` (`server/snapshots.ts` 24h loop) with Forgejo Actions scheduled workflows hitting authenticated endpoints. + +**New endpoints** (all behind `X-Cron-Secret` header): + +| Endpoint | Schedule | What it does | |---|---|---| -| Structured JSON logs to stdout, view via `docker logs brackt \| jq` | ~2 hours | Good enough for low-volume signals (tick drift, auto-pick latency). Start here. | -| Grafana + Loki + Promtail as sidecars in the compose | ~1 day | Real dashboards, log retention, query history | -| Hosted (Axiom / Better Stack / Grafana Cloud free tier) | ~half day | No infra to run; ship logs over the wire | +| `POST /admin/jobs/run-daily-snapshots` | `5 0 * * *` | Daily fantasy standings snapshots for active/draft seasons | +| `POST /admin/jobs/sync-and-simulate` | `0 */2 * * *` | Syncs standings from external APIs; simulates only when standings changed | +| `GET /healthz` | Traefik / Docker healthcheck | 200 `{ok:true}` when DB reachable, 503 otherwise | -**Recommendation:** start with structured JSON + `jq` for a week, escalate only if you find yourself wanting historical queries you can't get from logs. +**Schema additions** (migration 0118): `standingsLastChangedAt`, `lastSimulatedAt` on `sportsSeasons`. -**Why first:** every decision below is currently a guess. A week of metrics turns guesses into decisions, *and* gives the Phase 1 timer refactor a baseline to compare against. +**Change detection:** `syncStandings()` compares `gamesPlayed` + `leagueRank` against current DB rows before upsert; sets `standingsLastChangedAt` only on real change. Simulation skipped if nothing changed. + +**Setup required on production:** add `CRON_SECRET` to Forgejo repo secrets and to the server `.env`. --- -### Phase 1 — Deadline-based draft timer (3–5 days) ⭐ biggest win +### 🔜 Phase B — Production hardening + deploy automation (next, ~1 day) -**The problem:** the 1-second `setInterval` in `server/timer.ts` is the *cause* of nearly every downstream problem in this roadmap. Per active draft, every second it issues an UPDATE on `draftTimers`, several SELECTs, and a `timer-update` socket broadcast. It also makes the process un-killable without dropping drafts, makes multi-instance hard, and is the only thing in this stack that looks like it might need Go. +Three blockers for multi-replica work, plus automating compose file delivery. -**The insight:** the timer state is *deterministic from a deadline*. If you store `pickDeadlineAt` (absolute UTC) instead of `timeRemaining` (seconds), the remaining time at any moment is just `pickDeadlineAt − now()`. The server doesn't need to tick — it only needs to wake up *at the deadline* to run the auto-pick. +#### 1. Automate compose file delivery via deploy workflow -**The change:** +Currently `.production-info/docker-compose-production.yml` is a manual reference — the server's `~/brackt/docker-compose.yml` drifts silently. Fix: add an `appleboy/scp-action` step to the deploy job that copies the repo's compose file to the server on every deploy. -Schema (one new column total): -``` -seasons - + pickDeadlineAt timestamptz NULL -- the active deadline for the current pick; - -- NULL while paused, pre-draft, or completed -``` -Why on `seasons` and not `draftTimers`: only one team's deadline matters at a time (the current picker's). One row per draft matches the semantics and gives a trivial bootstrap query (`SELECT id FROM seasons WHERE status='draft' AND pickDeadlineAt IS NOT NULL`). The existing `draftTimers.timeRemaining` stays where it is — still needed for chess-clock bank tracking and as the "saved remaining seconds while paused" storage. +**Deploy workflow changes:** +- Add checkout step to the deploy job (needed to access the file) +- Add SCP step: copy `.production-info/docker-compose-production.yml` → `~/brackt/docker-compose.yml` +- Build with both `:latest` and `:{sha}` tags; pass `IMAGE_TAG=${{ github.sha }}` to the SSH step +- SSH step exports `IMAGE_TAG` before running `docker compose up` -Server behavior — funnel **every** deadline write through a single helper: -```ts -async function setNextPickDeadline(tx, { seasonId, teamId, budgetSeconds }) { - // 1. UPDATE seasons SET pickDeadlineAt = now() + budgetSeconds - // 2. UPDATE draftTimers SET timeRemaining = budgetSeconds (chess-clock: add increment) - // 3. (re)schedule in-memory setTimeout for this seasonId - // 4. emit { deadline, serverNow } on draft-${seasonId} -} -``` -Callsites (audit surface): -- **Draft auto-start** (`app/services/draft-autostart.ts`) — first pick's deadline. -- **Manual pick** (`app/routes/api/draft.make-pick.ts`) — next pick's deadline (currently writes `timeRemaining` only). -- **Auto-pick** (`server/timer.ts` rewrite) — next pick's deadline after auto-picking. -- **Admin time-bank adjust** (`app/routes/api/draft.adjust-time-bank.ts`) — recompute `pickDeadlineAt` to match new `timeRemaining`. -- **Resume from pause** — same as auto-start but for the current picker. +#### 2. Move secrets to a server-side `.env` file -Other behaviors: -- **Process restart:** on boot, query active drafts where `pickDeadlineAt IS NOT NULL` and re-schedule a `setTimeout` for each. Already-expired deadlines auto-pick immediately. -- **Pause:** save remaining into existing `draftTimers.timeRemaining` (`= pickDeadlineAt − now()`), null out `seasons.pickDeadlineAt`, clear the in-memory timeout. **No new "paused remaining" column needed** — `timeRemaining` already serves this purpose. -- **Overnight pause:** when scheduling, if the deadline falls inside the pause window, schedule for `resumeAtUTC` instead and emit a `frozen` flag. -- **`while_on` autodraft:** unchanged semantics — fire immediately on pick transition, capped at `totalTeams` iterations. The logic moves from the tick to the pick-completion handler (inside `setNextPickDeadline` or just before it). +Currently secrets reach the container via unknown means (no `env_file:` in compose, no `environment:` entries for DB credentials). Make this explicit: add `env_file: .env` to the brackt service; document all required vars in `.production-info/.env.production.example`. -Client behavior: -- Receive `{ deadline, serverNow }`; compute clock offset once. -- Render countdown locally from `deadline − clientNow() + offset`. Already has to do this between 1Hz emits today, so the interpolation code likely exists. +Non-secret config (NODE_ENV, PGBOUNCER, APP_URL, etc.) stays inline in the compose file. Secrets (DATABASE_URL, BETTER_AUTH_SECRET, CRON_SECRET, RESEND_API_KEY, Cloudinary, Discord, Google, Sentry, Turnstile) go in the server `.env`. -**Wins vs today:** +#### 3. Compose file fixes -| | 1s setInterval (today) | Deadline-based | -|---|---|---| -| DB writes per active draft per pick | ~30–120 | 1 | -| Socket emits per active draft per pick | ~30–120 | 1 (deadline) + 1 (pick made) | -| Server wake-ups per second | 1 always | 0 (sleeps until deadline) | -| Survives process restart | dies | one-line bootstrap query | -| Cross-process / multi-instance | hard (singleton tick required) | trivial (whichever instance handles the pick schedules the next deadline) | +- **Remove `container_name: brackt`** — Docker rejects duplicate names; this alone blocks `replicas: 2`. +- **Add healthcheck** using `/healthz` (already shipped in Phase A): + ```yaml + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:3000/healthz"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + ``` +- **Pin image tag** to `${CONTAINER_REGISTRY}/brackt:${IMAGE_TAG:-latest}` — rollback becomes `IMAGE_TAG= docker compose up`. -**Critical constraints:** -- **Single scheduler assumption.** Phase 1 runs in one process. Multi-instance leader-election (per-`seasonId` advisory lock + claim metadata) is deferred to Phase 5. Document this assumption explicitly in `server/timer.ts`. -- **Clock skew.** Send `serverNow` with each deadline event so clients can correct. -- **Reconnect path.** The loader must return `pickDeadlineAt` (and `serverNow`) so clients re-initializing after a reconnect render the correct countdown without waiting for a socket event. -- **Migration safety.** Run both systems in parallel for one draft (deadline-based writes the new column; old setInterval still drives picks). Compare. Flip when confident. +**Files:** +- `.production-info/docker-compose-production.yml` — remove container_name, add healthcheck, parameterise image tag, add env_file +- `.production-info/.env.production.example` — template for the server-side secrets file +- `.forgejo/workflows/deploy.yml` — add checkout + SCP step, build dual tags, pass IMAGE_TAG -**Why this matters more than splitting the tick into its own process:** it dissolves the problem rather than relocating it. Once the tick is deadline-based and idle 99% of the time, splitting it into its own container becomes a deploy-isolation play, not a correctness one. And the Phase 5 Go rewrite goes off the table permanently — there is no hot loop left to optimize. - -**Files:** `server/timer.ts` (rewrite as a scheduler module), `app/database/schema.ts` + new migration (single column on `seasons`), `app/models/draft-timer.ts` (new `setNextPickDeadline` helper — see implementation guide), `app/services/draft-autostart.ts` (call helper on draft start), `app/routes/api/draft.make-pick.ts` (call helper on manual pick), `app/routes/api/draft.adjust-time-bank.ts` (recompute deadline on adjust), `app/hooks/useDraftSocketEvents.ts` (sole `timer-update` consumer — switch to deadline event), draft room loader (`app/routes/leagues/$leagueId.draft.$seasonId.tsx`), socket event types. - -**See [`docs/agents/deadline-based-timer.md`](agents/deadline-based-timer.md) for the implementation guide** — schema, helper signature, callsite-by-callsite migration, test plan. - -**Gotchas to handle (load-bearing for correctness):** - -1. **Stale in-memory `setTimeout` after out-of-band DB writes.** Anything that mutates `pickDeadlineAt` outside the scheduler (admin extend, retro edit) leaves the captured closure stale. *Mitigation:* the auto-pick callback must re-read the row and verify `pickDeadlineAt` still equals the expected value before firing; if changed, reschedule and return. This single guard also handles wall-clock vs. monotonic drift (NTP jumps, VM resume): the re-check catches "deadline already passed" and "deadline moved" with the same code path. -2. **Pause-during-flight race.** `setTimeout` fires → enters auto-pick logic → admin pauses concurrently. The re-read in (1) must also check `draftPaused` before proceeding. Contract: in-flight picks complete; future scheduled picks don't fire after pause. -3. **Idempotency on restart.** Bootstrap query (`pickDeadlineAt IS NOT NULL`) fires all expired deadlines on boot. Idempotency is **already guaranteed** by `executeAutoPick` (`app/models/draft-utils.ts:499–512` pre-check + `:629–650` `ON CONFLICT DO NOTHING` on `(seasonId, pickNumber)`). Add an explicit test: "auto-pick is idempotent across process restart." -4. **Transaction boundary.** Pick row + next `pickDeadlineAt` write must share one transaction. Otherwise a crash between them leaves the draft "completed pick / no deadline" — the bootstrap query won't catch it. - -**Lower-severity edge cases (worth knowing, not blocking):** - -- **`setTimeout` 24.8-day overflow:** irrelevant for per-pick deadlines; *don't* reuse this primitive for `season.draftDateTime` auto-start — keep that on a low-frequency (30–60s) `setInterval` poll. -- **`while_on` cascade** moves from the tick to the pick-completion handler: check next team's autodraft mode before scheduling next deadline; fire immediately if `while_on`, capped at `totalTeams` iterations. -- **Client tab backgrounding** is *better* than today: browsers throttle timers, but `visibilitychange → visible` triggers a recompute from the current deadline, so countdowns snap to truth instead of waiting for the next server emit. -- **Clock skew:** server includes `serverNow` with every deadline event; client computes a one-time offset. -- **Migration of in-flight drafts:** two-stage deploy — first ship shadow writes alongside the existing tick (observe a day), then flip the authoritative path and drop the interval. -- **Multi-instance scheduling** (Phase 5 inheritance): defer the leader-election design; document the "single scheduler" assumption explicitly. The bootstrap query recovers a crashed scheduler on restart, which is sufficient for one instance. -- **Observability swap:** the Phase 0 "tick drift" metric becomes meaningless after Phase 1. Replace with **deadline-fire accuracy** = `actualFireTime − scheduledFireTime`. Same intent. +**Verification:** +- `curl https://brackt.com/healthz` → 200 `{ok:true}` +- Deploy with a SHA tag; confirm rollback by setting `IMAGE_TAG=` and rerunning compose +- `docker compose up --scale brackt=2` no longer fails with "name already in use" --- -### Phase 2 — Scheduled work: snapshots off `setInterval` (1 day) +### 🔜 Phase C — Zero-downtime deploys via Traefik sticky sessions (~2–3 days) -**The problem:** `server/snapshots.ts` uses a 24h `setInterval` that dies with the process, has no retry, and no visibility. As you add standings sync, scoring imports, EV snapshots, etc., this gets worse. +Requires Phase B (no `container_name`, healthcheck, pinned tag). -**The realistic options** (the original roadmap assumed pg-boss; given the production constraints, simpler choices win here): +Run 2 web containers behind Traefik with sticky sessions. Rolling deploy takes one replica out of rotation, upgrades it, puts it back — other replica keeps serving. -| Option | Pros | Cons | Verdict | +**Compose changes:** +```yaml +deploy: + replicas: 2 + +labels: + - "traefik.http.services.brackt.loadbalancer.sticky.cookie=true" + - "traefik.http.services.brackt.loadbalancer.sticky.cookie.name=brackt_instance" +``` + +Socket.IO clients must return to the same instance (sticky cookie). No Redis adapter needed at current scale — the second instance is purely for deploy rotation, not capacity. + +**Rolling deploy script:** +1. Scale down replica A (remove from Traefik pool) +2. Wait for drain (or 5s hard cut — reconnect works) +3. Pull new image on A, `docker compose up` A +4. Healthcheck passes → A back in pool +5. Repeat for B + +**Cross-instance broadcasts** (admin action on instance B needs to reach draft clients on instance A): use a pending-broadcasts table polled every 5s by each instance. With deadline-based timers the volume is tiny. Postgres `LISTEN/NOTIFY` is the upgrade path if polling proves too slow — requires a session-mode pgbouncer pool. + +**Files:** `.production-info/docker-compose-production.yml` (replicas + sticky labels), rolling deploy script. + +**Verification:** rolling-deploy both instances during a staging draft; clients see at most one brief reconnect. + +--- + +### Phase D — Optional: split tick into its own process (~1 week, not urgent) + +After Phase 1, the "tick" is an idle process that wakes only at pick deadlines. It can stay co-located with web indefinitely. Only revisit if you want to deploy web without touching timer code at all. + +**If/when:** extract `server/tick.ts` as a standalone entrypoint, run as a second compose service, enforce singleton via Postgres advisory lock. + +--- + +### Phase E — Defer: Socket.IO Redis adapter + +Only needed if a single draft's clients need to span multiple web instances (e.g., 50+ concurrent connections per draft, or geo-distributed). Not relevant at current scale. + +If/when: add Redis, `@socket.io/redis-adapter`, drop sticky sessions. ~1-day change if Phases B–C are clean. + +--- + +## Sequencing summary + +| Phase | Status | Effort | Key win | |---|---|---|---| -| **External HTTP cron** (healthchecks.io free / GitHub Actions `schedule:` / cron-job.org) hitting `POST /admin/jobs/run-daily-snapshots` | Zero new infra; dead-man-switch alerting built-in; sidesteps pgbouncer entirely | External dependency; needs auth on the endpoint | ⭐ start here | -| `node-cron` in a small worker container with Postgres advisory lock for singleton | Self-contained; trivial; no 3rd-party | Build basic retry/durability yourself; another container | Good middle ground | -| `pg-boss` | Real job system, retries, audit table | Uses `LISTEN/NOTIFY` and prepared statements — incompatible with pgbouncer transaction pooling. **On Vultr, this is solvable:** create a second session-mode pool just for pg-boss and pass its DSN to that workload only. No built-in UI. | Defer until you have 5+ scheduled jobs | -| `pg_cron` extension | Runs inside Postgres; rock solid; no scheduler process. **Supported on Vultr** (`CREATE EXTENSION IF NOT EXISTS pg_cron;`). | Logic lives in SQL; for app-level work you'd call out via `pg_net → POST /admin/jobs/snapshots`, which is just external HTTP cron with extra steps | Skip for one job; reconsider if you accumulate many | +| 1 | ✅ Done | — | Deadline-based timer; eliminates 1s tick | +| A | ✅ Done | — | External cron; automated standings sync + sim; /healthz | +| B | 🔜 Next | ~1d | Compose delivered by CI; pinned tags; healthcheck; unblocks C | +| C | Blocked on B | ~2–3d | Zero-downtime rolling deploys | +| D | Optional | ~1wk | Deploy web without restarting timer process | +| E | Deferred | ~1d | Cross-instance socket fan-out (not needed yet) | -**Recommendation:** external HTTP cron. The "job" becomes an authenticated `POST /admin/jobs/run-daily-snapshots` route that calls the existing `createSnapshotsForAllSeasons()`. Healthchecks.io's free tier gives you "you didn't run today" alerts for free. - -**Why now (not later):** it's a 1-hour change once you have an admin endpoint, and it removes the last `setInterval` from `server.ts` — which makes Phase 3 (process restart safety) trivially correct rather than "hopefully correct." - -**Files:** new admin route `app/routes/admin/jobs.run-daily-snapshots.tsx` (POST handler with shared-secret auth), remove `startSnapshotSystem()` call from `server.ts`, delete or repurpose `server/snapshots.ts`. - ---- - -### Phase 3 — Production hardening prerequisites (1–2 days) - -Before any multi-replica / zero-downtime work, the production compose needs three small fixes. These are cheap and worth doing regardless of what comes next. - -**The changes:** -1. **Remove hardcoded `container_name: brackt`.** Docker rejects duplicate names — this alone blocks scaling to 2 replicas. Use the service name + replica index (Compose handles this automatically when `container_name` is omitted). -2. **Add a healthcheck** to the brackt service (e.g., `GET /healthz` returns 200 once the server is listening and DB is reachable). Without this, Traefik will route to a half-started container during rolling deploys. -3. **Pin the image tag.** Move off `:latest` to a Git SHA or version. Rollbacks become a one-line edit; you can tell which replica is running which version during a rolling deploy. - -**Files:** `.production-info/docker-compose-production.yml`, new `app/routes/healthz.tsx`, deploy script (whatever sets the tag). - ---- - -### Phase 4 — Optional: split tick into its own process (1 week, no longer urgent) - -**Status after Phase 1:** the "tick" is now an idle process that wakes up at deadlines. It can be co-located with web indefinitely without correctness or performance issues. - -**When to revisit:** if you want pure deploy isolation — restart the web tier without touching the timer scheduler at all. This is a nice-to-have, not a blocker. - -**The change (if/when):** -- Extract timer scheduling into `server/tick.ts` standalone entrypoint. -- Run as a second container in compose. -- Tick still emits via Socket.IO — either via Postgres `LISTEN/NOTIFY` to the web tier (⚠️ verify pgbouncer mode supports session-scoped listeners) or via a pending-broadcasts table the web tier polls. With Phase 1 done, the broadcast volume is tiny — polling is fine. -- Singleton enforced via Postgres advisory lock (same primitive Phase 1 already uses for per-draft scheduling). - -**Why this is downgraded vs. the old roadmap:** Phase 1 already removed the per-second hot loop. The remaining motivation is purely "I want to deploy web independently of timer code." That's nice but not load-bearing. - ---- - -### Phase 5 — Zero-downtime web deploys via Traefik + sticky sessions (2–3 days) - -**The problem:** even after Phase 1, restarting the web container still drops sockets. - -**The change:** -- Run **2 web containers** behind Traefik (prerequisites from Phase 3 must be done first). -- Enable **sticky sessions** in Traefik via labels: `traefik.http.services.brackt.loadbalancer.sticky.cookie=true`. Socket.IO needs this — clients must return to the same instance until you add a pub/sub adapter. -- Rolling deploy: take instance A out of Traefik's pool → wait for connections to drain (or hard-cut, since reconnect works) → deploy A → put back in pool → repeat for B. -- This works **without Redis** because each draft room's clients are sticky to one instance. -- For broadcasts that need to reach *both* instances (e.g., timer scheduled on instance A, but admin action on instance B): use Postgres `LISTEN/NOTIFY` (⚠️ verify pgbouncer mode first) or a pending-broadcasts table. - -**Why this works at your scale:** with <20 concurrent drafts, a single web instance has zero capacity issues. The second instance is purely for deploy rotation. - -**Files:** `.production-info/docker-compose-production.yml` (replicas + sticky labels), deploy script. - ---- - -### Phase 6 — Defer: Socket.IO Redis adapter (only when needed) - -**When to revisit:** if you ever want a single draft's clients to span multiple web instances (e.g., 50+ concurrent connections per draft, or geo-distributed users). You're nowhere near that. - -**If/when you do:** add Redis, install `@socket.io/redis-adapter`, drop sticky sessions. ~1-day change if Phases 1–5 are clean. - ---- - -### ~~Phase 7 — Go tick rewrite~~ (removed) - -Phase 1 (deadline-based timer) eliminates the hot loop the Go rewrite was meant to optimize. The remaining scheduler is `setTimeout` and a Postgres query — Node handles this for free. - -If you want to learn Go, pick a greenfield side service (a webhook receiver for scoring feeds, for instance). Don't put it on the critical path of live drafts. - ---- - -## Sequencing Summary - -| # | Phase | Effort | Unblocks | -|---|---|---|---| -| 0 | Instrumentation | 1–2d | Every decision below | -| 1 | **Deadline-based draft timer** | 3–5d | Eliminates the 1s tick; makes everything else easier | -| 2 | Snapshots → external HTTP cron | 1d | Removes the last `setInterval`; clean restart semantics | -| 3 | Prod hardening (no `container_name`, healthcheck, pinned tag) | 1–2d | Required for any multi-replica work | -| 4 | Split tick into own process | ~1wk | Optional after Phase 1; deploy isolation only | -| 5 | 2 web instances + Traefik sticky sessions | 2–3d | Zero-downtime deploys (the actual goal) | -| 6 | Redis + socket.io adapter | *deferred* | Cross-instance fan-out (not needed at current scale) | - -**Total to "I can deploy without dropping connections": ~2 weeks of focused work** (Phases 0, 1, 2, 3, 5; Phase 4 optional). +**Total to zero-downtime deploys: ~3–4 days of focused work** (Phases B + C). --- ## Open questions -**Resolved (Vultr managed Postgres):** - -- **Pgbouncer modes:** Vultr ships pgbouncer built-in and supports all three modes (session, transaction, statement). You can create **multiple pools with different modes per cluster** via the Vultr panel — so the "pgbouncer incompatibility" concern for pg-boss / `LISTEN/NOTIFY` is *not* a hard blocker; it's "use a session-mode pool for those workloads, transaction-mode for the main app." *Action when relevant:* confirm what pool mode `PGBOUNCER=true` currently routes through (transaction is the typical default) and add a second session-mode pool the day you adopt pg-boss or `LISTEN/NOTIFY`. -- **`pg_cron`:** supported on Vultr (`CREATE EXTENSION IF NOT EXISTS pg_cron;`). Doesn't change the Phase 2 recommendation (external HTTP cron remains simpler for one job) but stays in the toolbox. - -**Still open:** - -- **Where do logs/metrics go?** Pick one in Phase 0 and commit. +- **Where do logs/metrics go?** Still unresolved. Options: structured JSON to stdout + `docker logs | jq` (zero effort), hosted (Better Stack / Axiom free tier, ~half day), or Grafana + Loki sidecars (~1 day). Pick one before adding more observability instrumentation. +- **Pgbouncer pool mode:** confirm current pool is transaction mode. Add a second session-mode pool before adopting `LISTEN/NOTIFY` or pg-boss. --- ## Critical files -- `server.ts` — main entrypoint; stops calling `startSnapshotSystem` (Phase 2); timer system becomes deadline-scheduler (Phase 1). -- `server/timer.ts` — rewrite to deadline-based scheduler (Phase 1); add tick drift metric *before* the rewrite to baseline (Phase 0). -- `server/snapshots.ts` — delete or repurpose; logic moves behind an admin endpoint (Phase 2). -- `server/socket.ts` — emit `{ deadline, serverNow }` events instead of per-second `timer-update` (Phase 1). -- `app/hooks/useDraftSocketEvents.ts` and draft room route — client-side countdown interpolation from deadline (Phase 1). -- `app/database/schema.ts` — add `pickDeadlineAt` to `seasons` (Phase 1). -- `.production-info/docker-compose-production.yml` — remove `container_name`, add healthcheck, pin tag, then add replicas + sticky labels (Phases 3, 5). -- New `app/routes/admin/jobs.run-daily-snapshots.tsx` — HTTP-triggered snapshot job (Phase 2). -- New `app/routes/healthz.tsx` — Traefik healthcheck target (Phase 3). - -## Verification - -- **Phase 0:** confirm metrics emit (grep logs for tick drift, auto-pick latency); leave running for a week before Phase 1 ships. -- **Phase 1:** run side-by-side mode in a staging draft — deadline-based writes shadow columns, setInterval still drives picks; confirm zero drift, then flip. Then: restart the process mid-draft, confirm timer keeps the right deadline and auto-pick fires at the original time. -- **Phase 2:** kill the web process mid-day; confirm the external cron still triggers tomorrow and the endpoint runs successfully. -- **Phase 3:** `curl /healthz` returns 200 only after DB is reachable; deploy with a pinned tag and confirm you can roll back by changing one line. -- **Phase 5:** rolling-deploy both web instances during a staging draft; users see at most a single brief reconnect (or none, with proper drain). +- `server/timer.ts` — deadline-based scheduler; `picksExpiresAt` on `draftTimers` +- `server/socket.ts` — emits `{ expiresAt, timeRemaining }` on `timer-pick-started` +- `app/routes/healthz.ts` — DB ping; used by Docker healthcheck and Traefik +- `app/routes/admin/jobs.run-daily-snapshots.ts` — daily snapshot cron endpoint +- `app/routes/admin/jobs.sync-and-simulate.ts` — standings sync + conditional sim cron endpoint +- `app/lib/cron-auth.ts` — `requireCronSecret()` helper for cron endpoints +- `app/services/standings-sync/index.ts` — `syncStandings()` returns `{ changed: boolean }` +- `.production-info/docker-compose-production.yml` — source of truth for production compose (Phase B: delivered by CI) +- `.production-info/.env.production.example` — template for server-side secrets (Phase B) +- `.forgejo/workflows/deploy.yml` — CI/CD pipeline (Phase B: add SCP + dual image tags) +- `.forgejo/workflows/daily-snapshots.yml` — daily snapshot cron (Forgejo Actions) +- `.forgejo/workflows/sync-and-simulate.yml` — 2h standings sync + sim cron (Forgejo Actions)