brackt/plans/zero-downtime-scaling.md

Plan: Zero-Downtime Deploys + Multi-Server Scaling

Context

Currently the draft timer (server/timer.ts) runs as a setInterval every second inside the web server process. This causes two problems:

1. Deploys interrupt active drafts — when the web server restarts during a rolling deploy, the timer stops, Socket.IO clients disconnect, and users see a "reconnecting" banner mid-draft.
2. Cannot run multiple web server instances — if two instances both run the timer tick, they independently decrement timeRemaining (2x speed) and both attempt to trigger autopicks on the same pick slot.

Additionally, server/snapshots.ts has the same per-instance setInterval problem (daily snapshots would be created multiple times).

There is no Redis, no distributed locking, and no graceful shutdown handler today.

Three Goals (in priority order)

1. Draft timer survives web server deploys — timer lives in a separate worker process that isn't restarted when the web server is.
2. Multiple web server instances can safely coexist — Socket.IO state shared via Redis adapter; timer runs only once (in the worker).
3. Cron jobs (standings sync, daily snapshots) run reliably once — managed inside the same worker.

---

Architecture Target

┌─────────────────────┐   Redis (pub/sub +     ┌─────────────────────┐
│   Web Server(s)     │   Socket.IO adapter)   │  Timer Worker       │
│  - HTTP / SSR       │◄──────────────────────►│  - Draft timers     │
│  - Socket.IO        │                         │  - Daily snapshots  │
│  - React Router     │                         │  - Standings crons  │
└─────────────────────┘                         └─────────────────────┘
         │                                               │
         └───────────────────── PostgreSQL ──────────────┘

Socket.IO uses @socket.io/redis-adapter so any instance can broadcast to any client room. The timer worker connects to the same Redis bus and emits into draft rooms — web servers relay those to connected clients.

---

Phase 1: Event-Driven Timer (stop per-second DB writes) ← START HERE ✅

Goal: replace the setInterval(1000) + per-second UPDATE draftTimers SET timeRemaining = GREATEST(timeRemaining - 1, 0) with a model where the server stores when the timer expires and uses a single setTimeout.

1a. DB schema change

Add two nullable columns to draft_timers:

// database/schema.ts
picksExpiresAt: timestamp("picks_expires_at"),  // null = paused or pick just made
picksStartedAt: timestamp("picks_started_at"),  // when current pick slot opened

Migration: npm run db:generate && npm run db:migrate

Keep timeRemaining — it is still the authoritative bank value in chess-clock mode and is needed when pausing/unpausing to know how much time to restore.

1b. Timer logic rewrite (server/timer.ts)

Replace setInterval(fn, 1000) with a scheduler that:

1. On startup, reads all active draft seasons from DB and calls schedulePick(season) for each.
2. schedulePick(season):
   • Looks up current team + their picksExpiresAt from DB.
   • If already set: setTimeout(onExpiry, picksExpiresAt - now).
   • If null (pick just started or timer not set): calls initPick(season) to write picksExpiresAt = now + timeRemaining and then schedules the timeout.
3. onExpiry(seasonId, teamId, pickNumber): triggers the autopick (same logic as today's triggerAutoPick). After pick resolves, calls schedulePick for the next team.
4. Recovery interval: a setInterval(recovery, 30_000) (not 1 s!) queries active drafts and re-calls schedulePick for any whose in-memory timeout was lost (e.g., process restart). This is the only periodic DB query.

Overnight pause handling: when an overnight pause window activates, cancel the in-memory setTimeout, set picksExpiresAt = null in DB, record remaining time in timeRemaining. When the window ends, restore via initPick. The recovery interval detects this automatically on restart.

Chess clock mode: timeRemaining stores the bank; picksExpiresAt = now + timeRemaining when a pick starts. picksStartedAt tracks when the pick opened. On pick completion, calculate actual elapsed: bankUsed = now - picksStartedAt, new timeRemaining = timeRemaining - bankUsed.

1c. Socket.IO event change

Stop emitting timer-update every second. Instead emit:

Event	When	Payload
timer-pick-started	Pick slot opens	{ seasonId, teamId, pickNumber, expiresAt, timeRemaining }
timer-paused	Draft paused or overnight pause	{ seasonId, teamId, overnightPauseActive, resumesAtUTC? }
timer-resumed	Pause lifted	{ seasonId, teamId, expiresAt, timeRemaining }
pick-made	Already exists	no change

Client (app/hooks/useDraftSocketEvents.ts) counts down locally from expiresAt using a client-side setInterval. Remove the server-driven timer display logic.

1d. Client update

• On timer-pick-started: store expiresAt in state, start a local setInterval to display countdown.
• On draft-state-sync (reconnect): server sends current picksExpiresAt so client can resume countdown.
• Remove any logic that relied on receiving timer-update every second.

Files changed in Phase 1

File	Change
database/schema.ts	Add picksExpiresAt, picksStartedAt to draftTimers
server/timer.ts	Rewrite: event-driven scheduler, 30s recovery interval
server/socket.d.ts	Add timer-pick-started, timer-paused, timer-resumed; remove timer-update
server/socket.ts	Emit new events on draft state sync (reconnect path)
app/hooks/useDraftSocketEvents.ts	Handle new events, remove timer-update handler
app/routes/leagues/$leagueId.draft.$seasonId.tsx	Client-side countdown from expiresAt

Verification:

• npm run typecheck — no errors
• npm run test:run — passes
• Manual: start a draft, watch picks countdown without seeing any "timer-update" Socket.IO events in browser devtools. Confirm pick fires at correct time. Confirm reconnect restores correct remaining time.

---

Phase 2: Separate Worker Process

Goal: timer runs in a process that is not restarted during web server deploys.

2a. Add Redis to infrastructure

Add to docker-compose.yml:

redis:
  image: redis:7-alpine
  restart: unless-stopped
  volumes:
    - redis_data:/data

Add env var REDIS_URL (default redis://localhost:6379).

Install: npm install ioredis @socket.io/redis-adapter

2b. Socket.IO Redis adapter on web servers

In server/socket.ts, after new Server(...):

import { createAdapter } from "@socket.io/redis-adapter";
import { createClient } from "ioredis";

const pub = createClient(process.env.REDIS_URL);
const sub = pub.duplicate();
io.adapter(createAdapter(pub, sub));

This means any server instance can broadcast to any client in any room.

Remove the connectedTeams in-memory Map — replace with Redis-backed presence or accept best-effort on reconnect.

2c. New worker entry point

Create worker/index.ts:

import { startDraftTimerSystem } from "../server/timer";
import { startSnapshotSystem } from "../server/snapshots";

startDraftTimerSystem();
startSnapshotSystem();

For Socket.IO emission from the worker: use direct Redis pub/sub (simpler than a second Socket.IO server). Add a server/timer-events.ts module with publishTimerEvent(channel, payload) (worker side) and subscribeTimerEvents(io) (web server side).

2d. Docker Compose

Add worker service:

worker:
  image: ${REGISTRY}/brackt:${TAG}
  command: node dist/worker.js
  depends_on: [db, redis]
  environment: *app-env
  restart: unless-stopped

Files to change in Phase 2

File	Change
docker-compose.yml	Add redis and worker services
server/socket.ts	Add Redis adapter; add timer-event subscription
server/timer.ts	Emit via Redis pub/sub instead of getSocketIO()
server/snapshots.ts	Move to worker only (remove import from server.ts)
worker/index.ts	New file: starts timer + snapshots
Dockerfile	Copy worker/ dir; add dist/worker.js output
package.json	Add build:worker script
.forgejo/workflows/deploy.yml	Deploy worker service alongside app

---

Phase 3: Zero-Downtime Web Server Deploy

Goal: rolling restart of web servers doesn't disconnect clients abruptly.

3a. Graceful shutdown

In server.ts, add SIGTERM handler:

process.on("SIGTERM", () => {
  httpServer.close(() => process.exit(0));
  io.close(() => process.exit(0));
  setTimeout(() => process.exit(0), 10_000);
});

3b. Health check endpoint

Add GET /health → { status: "ok" } to server/app.ts.

3c. Docker config

app:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
    interval: 10s
    start_period: 15s
  stop_grace_period: 15s

---

Phase 4: Cron Infrastructure in Worker

Goal: standings sync and other periodic tasks run on a schedule, once.

Install node-cron in the worker and schedule jobs:

cron.schedule("0 3 * * *", () => syncAllStandings(db));

Use a Postgres advisory lock if multiple workers are ever needed.

---

Implementation Order

Phase 1 (event-driven timer) — no infrastructure needed, pure code change
    ↓
Phase 2 (worker + Redis) — adds Redis, separates worker process
    ↓
Phase 3 (graceful shutdown + health check) — deploy strategy improvement
    ↓
Phase 4 (cron in worker) — scheduling infrastructure

Key Existing Code to Preserve/Reuse

Code	Path	Notes
startDraftTimerSystem() / stopDraftTimerSystem()	server/timer.ts	Keep the public API; replace internals
triggerAutoPick()	server/timer.ts	Unchanged logic, just called from setTimeout instead of setInterval
checkOvernightPause()	server/timer.ts	Keep as-is
executeAutoPick()	app/models/draft-utils.ts	Unchanged
getSocketIO()	server/socket.ts	Used in web server; worker will bypass via Redis pub/sub in Phase 2
draftSlotsCache	server/timer.ts	Keep — still valid optimization for the recovery check