brackt/plans/zero-downtime-scaling.md

# Plan: Zero-Downtime Deploys + Multi-Server Scaling

## Context

Currently the draft timer (`server/timer.ts`) runs as a `setInterval` every second inside the web server process. This causes two problems:

1. **Deploys interrupt active drafts** — when the web server restarts during a rolling deploy, the timer stops, Socket.IO clients disconnect, and users see a "reconnecting" banner mid-draft.
2. **Cannot run multiple web server instances** — if two instances both run the timer tick, they independently decrement `timeRemaining` (2x speed) and both attempt to trigger autopicks on the same pick slot.

Additionally, `server/snapshots.ts` has the same per-instance `setInterval` problem (daily snapshots would be created multiple times).

There is no Redis, no distributed locking, and no graceful shutdown handler today.

## Three Goals (in priority order)

1. **Draft timer survives web server deploys** — timer lives in a separate worker process that isn't restarted when the web server is.
2. **Multiple web server instances can safely coexist** — Socket.IO state shared via Redis adapter; timer runs only once (in the worker).
3. **Cron jobs (standings sync, daily snapshots) run reliably once** — managed inside the same worker.

---

## Architecture Target

```
┌─────────────────────┐   Redis (pub/sub +     ┌─────────────────────┐
│   Web Server(s)     │   Socket.IO adapter)   │  Timer Worker       │
│  - HTTP / SSR       │◄──────────────────────►│  - Draft timers     │
│  - Socket.IO        │                         │  - Daily snapshots  │
│  - React Router     │                         │  - Standings crons  │
└─────────────────────┘                         └─────────────────────┘
         │                                               │
         └───────────────────── PostgreSQL ──────────────┘
```

Socket.IO uses `@socket.io/redis-adapter` so any instance can broadcast to any client room. The timer worker connects to the same Redis bus and emits into draft rooms — web servers relay those to connected clients.

---

## Phase 1: Event-Driven Timer (stop per-second DB writes) ← START HERE ✅

**Goal**: replace the `setInterval(1000)` + per-second `UPDATE draftTimers SET timeRemaining = GREATEST(timeRemaining - 1, 0)` with a model where the server stores *when the timer expires* and uses a single `setTimeout`.

### 1a. DB schema change

Add two nullable columns to `draft_timers`:

```ts
// database/schema.ts
picksExpiresAt: timestamp("picks_expires_at"),  // null = paused or pick just made
picksStartedAt: timestamp("picks_started_at"),  // when current pick slot opened
```

**Migration**: `npm run db:generate && npm run db:migrate`

Keep `timeRemaining` — it is still the authoritative bank value in chess-clock mode and is needed when pausing/unpausing to know how much time to restore.

### 1b. Timer logic rewrite (`server/timer.ts`)

Replace `setInterval(fn, 1000)` with a scheduler that:

1. **On startup**, reads all active draft seasons from DB and calls `schedulePick(season)` for each.
2. **`schedulePick(season)`**:
   - Looks up current team + their `picksExpiresAt` from DB.
   - If already set: `setTimeout(onExpiry, picksExpiresAt - now)`.
   - If null (pick just started or timer not set): calls `initPick(season)` to write `picksExpiresAt = now + timeRemaining` and then schedules the timeout.
3. **`onExpiry(seasonId, teamId, pickNumber)`**: triggers the autopick (same logic as today's `triggerAutoPick`). After pick resolves, calls `schedulePick` for the next team.
4. **Recovery interval**: a `setInterval(recovery, 30_000)` (not 1 s!) queries active drafts and re-calls `schedulePick` for any whose in-memory timeout was lost (e.g., process restart). This is the only periodic DB query.

**Overnight pause handling**: when an overnight pause window activates, cancel the in-memory `setTimeout`, set `picksExpiresAt = null` in DB, record remaining time in `timeRemaining`. When the window ends, restore via `initPick`. The recovery interval detects this automatically on restart.

**Chess clock mode**: `timeRemaining` stores the bank; `picksExpiresAt = now + timeRemaining` when a pick starts. `picksStartedAt` tracks when the pick opened. On pick completion, calculate actual elapsed: `bankUsed = now - picksStartedAt`, new `timeRemaining = timeRemaining - bankUsed`.

### 1c. Socket.IO event change

Stop emitting `timer-update` every second. Instead emit:

| Event | When | Payload |
|---|---|---|
| `timer-pick-started` | Pick slot opens | `{ seasonId, teamId, pickNumber, expiresAt, timeRemaining }` |
| `timer-paused` | Draft paused or overnight pause | `{ seasonId, teamId, overnightPauseActive, resumesAtUTC? }` |
| `timer-resumed` | Pause lifted | `{ seasonId, teamId, expiresAt, timeRemaining }` |
| `pick-made` | Already exists | no change |

Client (`app/hooks/useDraftSocketEvents.ts`) counts down locally from `expiresAt` using a client-side `setInterval`. Remove the server-driven timer display logic.

### 1d. Client update

- On `timer-pick-started`: store `expiresAt` in state, start a local `setInterval` to display countdown.
- On `draft-state-sync` (reconnect): server sends current `picksExpiresAt` so client can resume countdown.
- Remove any logic that relied on receiving `timer-update` every second.

### Files changed in Phase 1

| File | Change |
|---|---|
| `database/schema.ts` | Add `picksExpiresAt`, `picksStartedAt` to `draftTimers` |
| `server/timer.ts` | Rewrite: event-driven scheduler, 30s recovery interval |
| `server/socket.d.ts` | Add `timer-pick-started`, `timer-paused`, `timer-resumed`; remove `timer-update` |
| `server/socket.ts` | Emit new events on draft state sync (reconnect path) |
| `app/hooks/useDraftSocketEvents.ts` | Handle new events, remove `timer-update` handler |
| `app/routes/leagues/$leagueId.draft.$seasonId.tsx` | Client-side countdown from `expiresAt` |

**Verification**:
- `npm run typecheck` — no errors
- `npm run test:run` — passes
- Manual: start a draft, watch picks countdown without seeing any "timer-update" Socket.IO events in browser devtools. Confirm pick fires at correct time. Confirm reconnect restores correct remaining time.

---

## Phase 2: Separate Worker Process

**Goal**: timer runs in a process that is not restarted during web server deploys.

### 2a. Add Redis to infrastructure

Add to `docker-compose.yml`:
```yaml
redis:
  image: redis:7-alpine
  restart: unless-stopped
  volumes:
    - redis_data:/data
```

Add env var `REDIS_URL` (default `redis://localhost:6379`).

Install: `npm install ioredis @socket.io/redis-adapter`

### 2b. Socket.IO Redis adapter on web servers

In `server/socket.ts`, after `new Server(...)`:
```ts
import { createAdapter } from "@socket.io/redis-adapter";
import { createClient } from "ioredis";

const pub = createClient(process.env.REDIS_URL);
const sub = pub.duplicate();
io.adapter(createAdapter(pub, sub));
```

This means any server instance can broadcast to any client in any room.

Remove the `connectedTeams` in-memory Map — replace with Redis-backed presence or accept best-effort on reconnect.

### 2c. New worker entry point

Create `worker/index.ts`:
```ts
import { startDraftTimerSystem } from "../server/timer";
import { startSnapshotSystem } from "../server/snapshots";

startDraftTimerSystem();
startSnapshotSystem();
```

**For Socket.IO emission from the worker**: use direct Redis pub/sub (simpler than a second Socket.IO server). Add a `server/timer-events.ts` module with `publishTimerEvent(channel, payload)` (worker side) and `subscribeTimerEvents(io)` (web server side).

### 2d. Docker Compose

Add worker service:
```yaml
worker:
  image: ${REGISTRY}/brackt:${TAG}
  command: node dist/worker.js
  depends_on: [db, redis]
  environment: *app-env
  restart: unless-stopped
```

### Files to change in Phase 2

| File | Change |
|---|---|
| `docker-compose.yml` | Add `redis` and `worker` services |
| `server/socket.ts` | Add Redis adapter; add timer-event subscription |
| `server/timer.ts` | Emit via Redis pub/sub instead of `getSocketIO()` |
| `server/snapshots.ts` | Move to worker only (remove import from `server.ts`) |
| `worker/index.ts` | New file: starts timer + snapshots |
| `Dockerfile` | Copy `worker/` dir; add `dist/worker.js` output |
| `package.json` | Add `build:worker` script |
| `.forgejo/workflows/deploy.yml` | Deploy worker service alongside app |

---

## Phase 3: Zero-Downtime Web Server Deploy

**Goal**: rolling restart of web servers doesn't disconnect clients abruptly.

### 3a. Graceful shutdown

In `server.ts`, add SIGTERM handler:
```ts
process.on("SIGTERM", () => {
  httpServer.close(() => process.exit(0));
  io.close(() => process.exit(0));
  setTimeout(() => process.exit(0), 10_000);
});
```

### 3b. Health check endpoint

Add `GET /health` → `{ status: "ok" }` to `server/app.ts`.

### 3c. Docker config

```yaml
app:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
    interval: 10s
    start_period: 15s
  stop_grace_period: 15s
```

---

## Phase 4: Cron Infrastructure in Worker

**Goal**: standings sync and other periodic tasks run on a schedule, once.

Install `node-cron` in the worker and schedule jobs:
```ts
cron.schedule("0 3 * * *", () => syncAllStandings(db));
```

Use a Postgres advisory lock if multiple workers are ever needed.

---

## Implementation Order

```
Phase 1 (event-driven timer) — no infrastructure needed, pure code change
    ↓
Phase 2 (worker + Redis) — adds Redis, separates worker process
    ↓
Phase 3 (graceful shutdown + health check) — deploy strategy improvement
    ↓
Phase 4 (cron in worker) — scheduling infrastructure
```

## Key Existing Code to Preserve/Reuse

| Code | Path | Notes |
|---|---|---|
| `startDraftTimerSystem()` / `stopDraftTimerSystem()` | `server/timer.ts` | Keep the public API; replace internals |
| `triggerAutoPick()` | `server/timer.ts` | Unchanged logic, just called from `setTimeout` instead of `setInterval` |
| `checkOvernightPause()` | `server/timer.ts` | Keep as-is |
| `executeAutoPick()` | `app/models/draft-utils.ts` | Unchanged |
| `getSocketIO()` | `server/socket.ts` | Used in web server; worker will bypass via Redis pub/sub in Phase 2 |
| `draftSlotsCache` | `server/timer.ts` | Keep — still valid optimization for the recovery check |