Spec and phased implementation plans for eliminating data duplication across overlapping qualifying-points windows (golf, tennis, CS2) by introducing canonical tournaments, participants, tournament_results, and surface-Elo tables above the existing per-window layer. - Phase 1a: rename existing per-window tables to season_* prefix - Phase 1b: create canonical tables + nullable FKs - Phase 2: backfill canonical from existing in-flight windows - Phase 3: cutover - sync service, admin UI, simulator read switch - Phase 4: cleanup - drop deprecated tables and routes Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
978 lines
29 KiB
Markdown
978 lines
29 KiB
Markdown
# Phase 2 — Backfill Canonical Layer Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Populate canonical tables (`tournaments`, `participants`, `tournament_results`, `participant_surface_elos`) from existing per-window rows, and set the FK columns (`scoring_events.tournament_id`, `season_participants.participant_id`). No runtime behavior changes; canonical tables become populated but are not yet read by any code path.
|
||
|
||
**Architecture:** One-off script with `--dry-run` and `--sport` flags. For each `sports_seasons` row with `scoring_pattern='qualifying_points'`, upsert canonical rows by `(sport_id, name, year)` for tournaments and `(sport_id, name)` for participants; copy completed `event_results` up to `tournament_results`; merge existing `season_participant_surface_elos` rows into canonical `participant_surface_elos`. Real run is preceded by an exact-match dry-run review.
|
||
|
||
**Tech Stack:** TypeScript, Drizzle ORM, PostgreSQL, Vitest + test DB.
|
||
|
||
**Prerequisite:** Phase 1b complete. Tag `phase-1b-complete` exists. Baseline fixtures in `test-fixtures/baselines/` captured in Phase 1a.
|
||
|
||
**Reference:** `docs/superpowers/specs/2026-05-01-canonical-tournament-layer-design.md` — Phase 2 section.
|
||
|
||
**Important constraints:**
|
||
- Do not copy `qualifying_points_awarded` from `event_results` to `tournament_results`. QP stays per-window.
|
||
- Do not write `participant_qualifying_totals`-equivalent data to canonical. QP totals stay per-window.
|
||
- Abort if two windows disagree on a canonical participant's surface Elo values (not expected today; fail loud).
|
||
|
||
---
|
||
|
||
### Task 1: Write the scoring-event → tournament matcher (pure function, TDD)
|
||
|
||
**Why:** The matcher has zero DB dependency; extract and test it in isolation first.
|
||
|
||
**Files:**
|
||
- Create: `scripts/backfill/match-tournament.ts`
|
||
- Create: `scripts/backfill/__tests__/match-tournament.test.ts`
|
||
|
||
- [ ] **Step 1: Write failing tests**
|
||
|
||
```typescript
|
||
// scripts/backfill/__tests__/match-tournament.test.ts
|
||
import { describe, it, expect } from "vitest";
|
||
import { extractTournamentIdentity } from "../match-tournament";
|
||
|
||
describe("extractTournamentIdentity", () => {
|
||
it("extracts (name, year) from a scoring event for golf", () => {
|
||
const got = extractTournamentIdentity({
|
||
name: "Masters Tournament",
|
||
eventDate: "2026-04-09",
|
||
eventType: "major_tournament",
|
||
});
|
||
expect(got).toEqual({ name: "Masters Tournament", year: 2026 });
|
||
});
|
||
|
||
it("uses eventDate year when event name doesn't embed year", () => {
|
||
const got = extractTournamentIdentity({
|
||
name: "Wimbledon",
|
||
eventDate: "2026-07-01",
|
||
eventType: "major_tournament",
|
||
});
|
||
expect(got).toEqual({ name: "Wimbledon", year: 2026 });
|
||
});
|
||
|
||
it("strips embedded year from name if present", () => {
|
||
const got = extractTournamentIdentity({
|
||
name: "Wimbledon 2026",
|
||
eventDate: "2026-07-01",
|
||
eventType: "major_tournament",
|
||
});
|
||
expect(got).toEqual({ name: "Wimbledon", year: 2026 });
|
||
});
|
||
|
||
it("throws if eventDate is null and name has no year", () => {
|
||
expect(() =>
|
||
extractTournamentIdentity({
|
||
name: "Wimbledon",
|
||
eventDate: null,
|
||
eventType: "major_tournament",
|
||
}),
|
||
).toThrow(/cannot determine year/i);
|
||
});
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run failing test**
|
||
|
||
Run: `npm run test:run -- scripts/backfill/__tests__/match-tournament.test.ts`
|
||
Expected: FAIL — module missing.
|
||
|
||
- [ ] **Step 3: Implement the pure function**
|
||
|
||
```typescript
|
||
// scripts/backfill/match-tournament.ts
|
||
interface ScoringEventInput {
|
||
name: string;
|
||
eventDate: string | null;
|
||
eventType: string;
|
||
}
|
||
|
||
export interface TournamentIdentity {
|
||
name: string;
|
||
year: number;
|
||
}
|
||
|
||
const YEAR_SUFFIX = /\s+(\d{4})\s*$/;
|
||
|
||
export function extractTournamentIdentity(ev: ScoringEventInput): TournamentIdentity {
|
||
const trimmed = ev.name.trim();
|
||
const yearMatch = trimmed.match(YEAR_SUFFIX);
|
||
let cleanName = trimmed;
|
||
let year: number | null = null;
|
||
|
||
if (yearMatch) {
|
||
year = parseInt(yearMatch[1], 10);
|
||
cleanName = trimmed.replace(YEAR_SUFFIX, "").trim();
|
||
}
|
||
|
||
if (year === null && ev.eventDate) {
|
||
year = parseInt(ev.eventDate.slice(0, 4), 10);
|
||
}
|
||
|
||
if (year === null) {
|
||
throw new Error(
|
||
`cannot determine year for event "${ev.name}" — no year in name and eventDate is null`,
|
||
);
|
||
}
|
||
|
||
return { name: cleanName, year };
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Verify tests pass**
|
||
|
||
Run: `npm run test:run -- scripts/backfill/__tests__/match-tournament.test.ts`
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add scripts/backfill/
|
||
git commit -m "feat(backfill): tournament identity extraction from scoring events"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: Write the backfill orchestrator with dry-run (TDD against test DB)
|
||
|
||
**Files:**
|
||
- Create: `scripts/backfill-canonical-layer.ts`
|
||
- Create: `scripts/__tests__/backfill-canonical-layer.test.ts`
|
||
|
||
- [ ] **Step 1: Write the test — empty DB no-op**
|
||
|
||
```typescript
|
||
// scripts/__tests__/backfill-canonical-layer.test.ts
|
||
import { describe, it, expect, beforeEach } from "vitest";
|
||
import { db } from "~/db/context";
|
||
import {
|
||
tournaments,
|
||
participants,
|
||
tournamentResults,
|
||
participantSurfaceElos,
|
||
scoringEvents,
|
||
seasonParticipants,
|
||
sportsSeasons,
|
||
sports,
|
||
eventResults,
|
||
seasonParticipantSurfaceElos,
|
||
} from "database/schema";
|
||
import { runBackfill } from "../backfill-canonical-layer";
|
||
|
||
async function truncateAll() {
|
||
await db.delete(tournamentResults);
|
||
await db.delete(participantSurfaceElos);
|
||
await db.delete(participants);
|
||
await db.delete(tournaments);
|
||
await db.delete(eventResults);
|
||
await db.delete(seasonParticipantSurfaceElos);
|
||
await db.delete(seasonParticipants);
|
||
await db.delete(scoringEvents);
|
||
await db.delete(sportsSeasons);
|
||
await db.delete(sports);
|
||
}
|
||
|
||
describe("backfill-canonical-layer", () => {
|
||
beforeEach(async () => {
|
||
await truncateAll();
|
||
});
|
||
|
||
it("no-ops on empty DB", async () => {
|
||
const result = await runBackfill({ dryRun: false });
|
||
expect(result.tournamentsCreated).toBe(0);
|
||
expect(result.participantsCreated).toBe(0);
|
||
expect(result.tournamentResultsCreated).toBe(0);
|
||
expect(result.surfaceElosCreated).toBe(0);
|
||
});
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run failing test**
|
||
|
||
Expected: FAIL (module missing).
|
||
|
||
- [ ] **Step 3: Implement skeleton**
|
||
|
||
```typescript
|
||
// scripts/backfill-canonical-layer.ts
|
||
import { db } from "~/db/context";
|
||
import {
|
||
tournaments,
|
||
participants,
|
||
tournamentResults,
|
||
participantSurfaceElos,
|
||
scoringEvents,
|
||
seasonParticipants,
|
||
sportsSeasons,
|
||
eventResults,
|
||
seasonParticipantSurfaceElos,
|
||
} from "database/schema";
|
||
import { eq, and, isNull } from "drizzle-orm";
|
||
import { extractTournamentIdentity } from "./backfill/match-tournament";
|
||
|
||
export interface BackfillOptions {
|
||
dryRun: boolean;
|
||
sportId?: string;
|
||
}
|
||
|
||
export interface BackfillReport {
|
||
tournamentsCreated: number;
|
||
tournamentsLinked: number; // scoringEvents with tournament_id now set
|
||
participantsCreated: number;
|
||
participantsLinked: number; // seasonParticipants with participant_id now set
|
||
tournamentResultsCreated: number;
|
||
surfaceElosCreated: number;
|
||
warnings: string[];
|
||
errors: string[];
|
||
}
|
||
|
||
export async function runBackfill(opts: BackfillOptions): Promise<BackfillReport> {
|
||
const report: BackfillReport = {
|
||
tournamentsCreated: 0,
|
||
tournamentsLinked: 0,
|
||
participantsCreated: 0,
|
||
participantsLinked: 0,
|
||
tournamentResultsCreated: 0,
|
||
surfaceElosCreated: 0,
|
||
warnings: [],
|
||
errors: [],
|
||
};
|
||
|
||
// Eligible sports seasons
|
||
const seasons = await db
|
||
.select()
|
||
.from(sportsSeasons)
|
||
.where(eq(sportsSeasons.scoringPattern, "qualifying_points"));
|
||
|
||
for (const season of seasons) {
|
||
if (opts.sportId && season.sportId !== opts.sportId) continue;
|
||
await backfillSeason(season, opts, report);
|
||
}
|
||
|
||
return report;
|
||
}
|
||
|
||
async function backfillSeason(
|
||
season: typeof sportsSeasons.$inferSelect,
|
||
opts: BackfillOptions,
|
||
report: BackfillReport,
|
||
): Promise<void> {
|
||
// Implemented in later tasks.
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Verify empty-DB test passes**
|
||
|
||
Run: `npm run test:run -- scripts/__tests__/backfill-canonical-layer.test.ts`
|
||
Expected: PASS (empty DB no-op).
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add scripts/backfill-canonical-layer.ts scripts/__tests__/backfill-canonical-layer.test.ts
|
||
git commit -m "feat(backfill): orchestrator skeleton with empty-DB test"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Tournament + scoring-event linking
|
||
|
||
**Files:**
|
||
- Modify: `scripts/backfill-canonical-layer.ts`
|
||
- Modify: `scripts/__tests__/backfill-canonical-layer.test.ts`
|
||
|
||
- [ ] **Step 1: Add failing test — golf window with 4 events**
|
||
|
||
Append to the test file:
|
||
|
||
```typescript
|
||
async function seedGolfWindowWith4Events() {
|
||
const [sport] = await db
|
||
.insert(sports)
|
||
.values({ name: "Golf", type: "individual", simulatorType: "golf_qualifying_points" })
|
||
.returning();
|
||
|
||
const [ss] = await db
|
||
.insert(sportsSeasons)
|
||
.values({
|
||
sportId: sport.id,
|
||
year: 2026,
|
||
scoringPattern: "qualifying_points",
|
||
totalMajors: 4,
|
||
majorsCompleted: 1,
|
||
})
|
||
.returning();
|
||
|
||
const events = await db
|
||
.insert(scoringEvents)
|
||
.values([
|
||
{ sportsSeasonId: ss.id, name: "Masters Tournament", eventDate: "2026-04-09", eventType: "major_tournament", isQualifyingEvent: true },
|
||
{ sportsSeasonId: ss.id, name: "PGA Championship", eventDate: "2026-05-14", eventType: "major_tournament", isQualifyingEvent: true },
|
||
{ sportsSeasonId: ss.id, name: "US Open", eventDate: "2026-06-18", eventType: "major_tournament", isQualifyingEvent: true },
|
||
{ sportsSeasonId: ss.id, name: "The Open Championship", eventDate: "2026-07-16", eventType: "major_tournament", isQualifyingEvent: true },
|
||
])
|
||
.returning();
|
||
|
||
return { sport, ss, events };
|
||
}
|
||
|
||
it("creates tournaments and links scoringEvents for a golf window", async () => {
|
||
const { sport, events } = await seedGolfWindowWith4Events();
|
||
const report = await runBackfill({ dryRun: false });
|
||
|
||
expect(report.tournamentsCreated).toBe(4);
|
||
expect(report.tournamentsLinked).toBe(4);
|
||
|
||
const ts = await db.select().from(tournaments).where(eq(tournaments.sportId, sport.id));
|
||
expect(ts.map((t) => t.name).sort()).toEqual(
|
||
["Masters Tournament", "PGA Championship", "The Open Championship", "US Open"],
|
||
);
|
||
expect(ts.every((t) => t.year === 2026)).toBe(true);
|
||
|
||
const linkedEvents = await db.select().from(scoringEvents).where(eq(scoringEvents.sportsSeasonId, events[0].sportsSeasonId));
|
||
expect(linkedEvents.every((e) => e.tournamentId !== null)).toBe(true);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run failing test**
|
||
|
||
Expected: FAIL (nothing is created yet).
|
||
|
||
- [ ] **Step 3: Implement tournament backfill in `backfillSeason`**
|
||
|
||
```typescript
|
||
async function backfillSeason(season, opts, report) {
|
||
const events = await db
|
||
.select()
|
||
.from(scoringEvents)
|
||
.where(eq(scoringEvents.sportsSeasonId, season.id));
|
||
|
||
for (const ev of events) {
|
||
if (ev.tournamentId) continue; // already linked
|
||
|
||
let identity;
|
||
try {
|
||
identity = extractTournamentIdentity({
|
||
name: ev.name,
|
||
eventDate: ev.eventDate,
|
||
eventType: ev.eventType,
|
||
});
|
||
} catch (e: any) {
|
||
report.errors.push(`event ${ev.id}: ${e.message}`);
|
||
continue;
|
||
}
|
||
|
||
const existing = await db
|
||
.select()
|
||
.from(tournaments)
|
||
.where(
|
||
and(
|
||
eq(tournaments.sportId, season.sportId),
|
||
eq(tournaments.name, identity.name),
|
||
eq(tournaments.year, identity.year),
|
||
),
|
||
);
|
||
|
||
let tournamentId: string;
|
||
if (existing.length > 0) {
|
||
tournamentId = existing[0].id;
|
||
} else {
|
||
if (!opts.dryRun) {
|
||
const [row] = await db
|
||
.insert(tournaments)
|
||
.values({
|
||
sportId: season.sportId,
|
||
name: identity.name,
|
||
year: identity.year,
|
||
startsAt: ev.eventDate ? new Date(ev.eventDate) : null,
|
||
status: ev.eventDate && new Date(ev.eventDate) < new Date() ? "completed" : "scheduled",
|
||
})
|
||
.returning();
|
||
tournamentId = row.id;
|
||
} else {
|
||
tournamentId = "00000000-0000-0000-0000-000000000000"; // placeholder
|
||
}
|
||
report.tournamentsCreated += 1;
|
||
}
|
||
|
||
if (!opts.dryRun) {
|
||
await db
|
||
.update(scoringEvents)
|
||
.set({ tournamentId })
|
||
.where(eq(scoringEvents.id, ev.id));
|
||
}
|
||
report.tournamentsLinked += 1;
|
||
}
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run tests**
|
||
|
||
Run: `npm run test:run -- scripts/__tests__/backfill-canonical-layer.test.ts`
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add scripts/
|
||
git commit -m "feat(backfill): link scoring events to canonical tournaments"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: Participant backfill + linking
|
||
|
||
**Files:**
|
||
- Modify: `scripts/backfill-canonical-layer.ts`
|
||
- Modify: `scripts/__tests__/backfill-canonical-layer.test.ts`
|
||
|
||
- [ ] **Step 1: Add failing test**
|
||
|
||
```typescript
|
||
it("creates canonical participants and links season_participants", async () => {
|
||
const { sport, ss } = await seedGolfWindowWith4Events();
|
||
const [sp1] = await db
|
||
.insert(seasonParticipants)
|
||
.values({ sportsSeasonId: ss.id, name: "Rory McIlroy" })
|
||
.returning();
|
||
const [sp2] = await db
|
||
.insert(seasonParticipants)
|
||
.values({ sportsSeasonId: ss.id, name: "Scottie Scheffler" })
|
||
.returning();
|
||
|
||
const report = await runBackfill({ dryRun: false });
|
||
expect(report.participantsCreated).toBe(2);
|
||
expect(report.participantsLinked).toBe(2);
|
||
|
||
const canonical = await db.select().from(participants).where(eq(participants.sportId, sport.id));
|
||
expect(canonical.map((p) => p.name).sort()).toEqual(["Rory McIlroy", "Scottie Scheffler"]);
|
||
|
||
const [relinkedSP] = await db.select().from(seasonParticipants).where(eq(seasonParticipants.id, sp1.id));
|
||
expect(relinkedSP.participantId).toBe(canonical.find((p) => p.name === "Rory McIlroy")!.id);
|
||
});
|
||
|
||
it("re-running the backfill does not create duplicate canonical participants", async () => {
|
||
const { ss } = await seedGolfWindowWith4Events();
|
||
await db.insert(seasonParticipants).values({ sportsSeasonId: ss.id, name: "Jon Rahm" });
|
||
|
||
await runBackfill({ dryRun: false });
|
||
await runBackfill({ dryRun: false });
|
||
|
||
const rows = await db.select().from(participants);
|
||
expect(rows.filter((r) => r.name === "Jon Rahm")).toHaveLength(1);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run failing tests**
|
||
|
||
Expected: FAIL.
|
||
|
||
- [ ] **Step 3: Extend `backfillSeason` with participant logic**
|
||
|
||
Append inside `backfillSeason` (after the event loop):
|
||
|
||
```typescript
|
||
const seasonPs = await db
|
||
.select()
|
||
.from(seasonParticipants)
|
||
.where(eq(seasonParticipants.sportsSeasonId, season.id));
|
||
|
||
for (const sp of seasonPs) {
|
||
if (sp.participantId) continue;
|
||
|
||
const existing = await db
|
||
.select()
|
||
.from(participants)
|
||
.where(and(eq(participants.sportId, season.sportId), eq(participants.name, sp.name)));
|
||
|
||
let canonicalId: string;
|
||
if (existing.length > 0) {
|
||
canonicalId = existing[0].id;
|
||
} else {
|
||
if (!opts.dryRun) {
|
||
const [row] = await db
|
||
.insert(participants)
|
||
.values({ sportId: season.sportId, name: sp.name })
|
||
.returning();
|
||
canonicalId = row.id;
|
||
} else {
|
||
canonicalId = "00000000-0000-0000-0000-000000000000";
|
||
}
|
||
report.participantsCreated += 1;
|
||
}
|
||
|
||
if (!opts.dryRun) {
|
||
await db
|
||
.update(seasonParticipants)
|
||
.set({ participantId: canonicalId })
|
||
.where(eq(seasonParticipants.id, sp.id));
|
||
}
|
||
report.participantsLinked += 1;
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run tests**
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add scripts/
|
||
git commit -m "feat(backfill): canonical participants + season-participant linking"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Copy completed event results to tournament_results (Masters 2026 case)
|
||
|
||
**Files:**
|
||
- Modify: `scripts/backfill-canonical-layer.ts`
|
||
- Modify: `scripts/__tests__/backfill-canonical-layer.test.ts`
|
||
|
||
- [ ] **Step 1: Add failing test — Masters 2026 round-trip**
|
||
|
||
```typescript
|
||
it("copies completed event_results to tournament_results without touching QP", async () => {
|
||
const { ss, events } = await seedGolfWindowWith4Events();
|
||
const masters = events.find((e) => e.name === "Masters Tournament")!;
|
||
const [sp] = await db
|
||
.insert(seasonParticipants)
|
||
.values({ sportsSeasonId: ss.id, name: "Scottie Scheffler" })
|
||
.returning();
|
||
|
||
await db.insert(eventResults).values({
|
||
scoringEventId: masters.id,
|
||
seasonParticipantId: sp.id,
|
||
placement: 1,
|
||
rawScore: "-10",
|
||
qualifyingPointsAwarded: "100.00",
|
||
});
|
||
|
||
const report = await runBackfill({ dryRun: false });
|
||
expect(report.tournamentResultsCreated).toBe(1);
|
||
|
||
const trs = await db.select().from(tournamentResults);
|
||
expect(trs).toHaveLength(1);
|
||
expect(trs[0].placement).toBe(1);
|
||
expect(trs[0].rawScore).toBe("-10");
|
||
|
||
// QP on event_results must be untouched
|
||
const [er] = await db.select().from(eventResults).where(eq(eventResults.scoringEventId, masters.id));
|
||
expect(er.qualifyingPointsAwarded).toBe("100.00");
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run failing test**
|
||
|
||
Expected: FAIL.
|
||
|
||
- [ ] **Step 3: Extend `backfillSeason`**
|
||
|
||
Append inside `backfillSeason` after participant linking:
|
||
|
||
```typescript
|
||
// Refresh events to get tournament_ids set earlier in this loop
|
||
const refreshedEvents = await db
|
||
.select()
|
||
.from(scoringEvents)
|
||
.where(eq(scoringEvents.sportsSeasonId, season.id));
|
||
|
||
for (const ev of refreshedEvents) {
|
||
if (!ev.tournamentId) continue;
|
||
const ers = await db
|
||
.select()
|
||
.from(eventResults)
|
||
.where(eq(eventResults.scoringEventId, ev.id));
|
||
|
||
for (const er of ers) {
|
||
// Find canonical participant via the season participant
|
||
const [sp] = await db
|
||
.select()
|
||
.from(seasonParticipants)
|
||
.where(eq(seasonParticipants.id, er.seasonParticipantId));
|
||
if (!sp || !sp.participantId) continue;
|
||
|
||
// Skip the zero-QP filler rows (they carry no new information)
|
||
if (er.placement === null && er.rawScore === null) continue;
|
||
|
||
const existing = await db
|
||
.select()
|
||
.from(tournamentResults)
|
||
.where(
|
||
and(
|
||
eq(tournamentResults.tournamentId, ev.tournamentId),
|
||
eq(tournamentResults.participantId, sp.participantId),
|
||
),
|
||
);
|
||
|
||
if (existing.length > 0) continue;
|
||
|
||
if (!opts.dryRun) {
|
||
await db.insert(tournamentResults).values({
|
||
tournamentId: ev.tournamentId,
|
||
participantId: sp.participantId,
|
||
placement: er.placement,
|
||
rawScore: er.rawScore,
|
||
});
|
||
}
|
||
report.tournamentResultsCreated += 1;
|
||
}
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run tests**
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add scripts/
|
||
git commit -m "feat(backfill): copy completed event_results into tournament_results"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Surface Elo backfill (with conflict detection)
|
||
|
||
**Files:**
|
||
- Modify: `scripts/backfill-canonical-layer.ts`
|
||
- Modify: `scripts/__tests__/backfill-canonical-layer.test.ts`
|
||
|
||
- [ ] **Step 1: Add failing tests**
|
||
|
||
```typescript
|
||
async function seedTennisWithSurfaceElo() {
|
||
const [sport] = await db
|
||
.insert(sports)
|
||
.values({ name: "Tennis", type: "individual", simulatorType: "tennis_qualifying_points" })
|
||
.returning();
|
||
const [ss] = await db
|
||
.insert(sportsSeasons)
|
||
.values({ sportId: sport.id, year: 2026, scoringPattern: "qualifying_points", totalMajors: 4 })
|
||
.returning();
|
||
const [sp] = await db
|
||
.insert(seasonParticipants)
|
||
.values({ sportsSeasonId: ss.id, name: "Novak Djokovic" })
|
||
.returning();
|
||
await db
|
||
.insert(seasonParticipantSurfaceElos)
|
||
.values({
|
||
participantId: sp.id,
|
||
sportsSeasonId: ss.id,
|
||
eloHard: 2100,
|
||
eloClay: 2000,
|
||
eloGrass: 2050,
|
||
worldRanking: 3,
|
||
});
|
||
return { sport, ss, sp };
|
||
}
|
||
|
||
it("backfills canonical surface elo from the single per-window row", async () => {
|
||
const { sport } = await seedTennisWithSurfaceElo();
|
||
const report = await runBackfill({ dryRun: false });
|
||
|
||
expect(report.surfaceElosCreated).toBe(1);
|
||
const canonicalPs = await db.select().from(participants).where(eq(participants.sportId, sport.id));
|
||
const [canonicalElo] = await db
|
||
.select()
|
||
.from(participantSurfaceElos)
|
||
.where(eq(participantSurfaceElos.participantId, canonicalPs[0].id));
|
||
expect(canonicalElo.eloHard).toBe(2100);
|
||
expect(canonicalElo.eloClay).toBe(2000);
|
||
expect(canonicalElo.eloGrass).toBe(2050);
|
||
expect(canonicalElo.worldRanking).toBe(3);
|
||
});
|
||
|
||
it("aborts if two windows disagree on a canonical participant's surface elo", async () => {
|
||
const { sport, ss, sp } = await seedTennisWithSurfaceElo();
|
||
|
||
// Second window with same canonical participant name but different elo
|
||
const [ss2] = await db
|
||
.insert(sportsSeasons)
|
||
.values({ sportId: sport.id, year: 2027, scoringPattern: "qualifying_points", totalMajors: 4 })
|
||
.returning();
|
||
const [sp2] = await db
|
||
.insert(seasonParticipants)
|
||
.values({ sportsSeasonId: ss2.id, name: "Novak Djokovic" })
|
||
.returning();
|
||
await db.insert(seasonParticipantSurfaceElos).values({
|
||
participantId: sp2.id,
|
||
sportsSeasonId: ss2.id,
|
||
eloHard: 2099, // different
|
||
eloClay: 2000,
|
||
eloGrass: 2050,
|
||
});
|
||
|
||
const report = await runBackfill({ dryRun: false });
|
||
expect(report.errors.some((e) => /conflict/i.test(e))).toBe(true);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Run failing tests**
|
||
|
||
Expected: FAIL.
|
||
|
||
- [ ] **Step 3: Extend `backfillSeason`**
|
||
|
||
Append to `backfillSeason`:
|
||
|
||
```typescript
|
||
const elos = await db
|
||
.select()
|
||
.from(seasonParticipantSurfaceElos)
|
||
.where(eq(seasonParticipantSurfaceElos.sportsSeasonId, season.id));
|
||
|
||
for (const row of elos) {
|
||
const [sp] = await db
|
||
.select()
|
||
.from(seasonParticipants)
|
||
.where(eq(seasonParticipants.id, row.participantId));
|
||
if (!sp || !sp.participantId) continue;
|
||
|
||
const [existing] = await db
|
||
.select()
|
||
.from(participantSurfaceElos)
|
||
.where(eq(participantSurfaceElos.participantId, sp.participantId));
|
||
|
||
if (existing) {
|
||
const mismatched =
|
||
existing.eloHard !== row.eloHard ||
|
||
existing.eloClay !== row.eloClay ||
|
||
existing.eloGrass !== row.eloGrass ||
|
||
existing.worldRanking !== row.worldRanking;
|
||
if (mismatched) {
|
||
report.errors.push(
|
||
`surface-elo conflict for canonical participant ${sp.participantId} (name=${sp.name})`,
|
||
);
|
||
}
|
||
continue;
|
||
}
|
||
|
||
if (!opts.dryRun) {
|
||
await db.insert(participantSurfaceElos).values({
|
||
participantId: sp.participantId,
|
||
eloHard: row.eloHard,
|
||
eloClay: row.eloClay,
|
||
eloGrass: row.eloGrass,
|
||
worldRanking: row.worldRanking,
|
||
});
|
||
}
|
||
report.surfaceElosCreated += 1;
|
||
}
|
||
```
|
||
|
||
- [ ] **Step 4: Run tests**
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add scripts/
|
||
git commit -m "feat(backfill): canonical surface-elo with conflict detection"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: CLI wrapper + dry-run output
|
||
|
||
**Files:**
|
||
- Create: `scripts/backfill-cli.ts` (CLI entry)
|
||
- Modify: `scripts/backfill-canonical-layer.ts` (optional — if the main file exported a report shape, we format it here)
|
||
|
||
- [ ] **Step 1: Write the CLI**
|
||
|
||
```typescript
|
||
// scripts/backfill-cli.ts
|
||
import { runBackfill, BackfillOptions } from "./backfill-canonical-layer";
|
||
|
||
function parseArgs(): BackfillOptions {
|
||
const args = process.argv.slice(2);
|
||
const opts: BackfillOptions = { dryRun: true };
|
||
for (const a of args) {
|
||
if (a === "--apply") opts.dryRun = false;
|
||
else if (a === "--dry-run") opts.dryRun = true;
|
||
else if (a.startsWith("--sport=")) opts.sportId = a.slice("--sport=".length);
|
||
else if (a === "--help") {
|
||
console.log("Usage: tsx scripts/backfill-cli.ts [--dry-run|--apply] [--sport=<uuid>]");
|
||
process.exit(0);
|
||
} else {
|
||
console.error(`unknown arg: ${a}`);
|
||
process.exit(1);
|
||
}
|
||
}
|
||
return opts;
|
||
}
|
||
|
||
async function main() {
|
||
const opts = parseArgs();
|
||
console.log(`Running backfill (dryRun=${opts.dryRun}, sportId=${opts.sportId ?? "all"})`);
|
||
const report = await runBackfill(opts);
|
||
|
||
console.log("---");
|
||
console.log(`tournamentsCreated: ${report.tournamentsCreated}`);
|
||
console.log(`tournamentsLinked: ${report.tournamentsLinked}`);
|
||
console.log(`participantsCreated: ${report.participantsCreated}`);
|
||
console.log(`participantsLinked: ${report.participantsLinked}`);
|
||
console.log(`tournamentResultsCreated: ${report.tournamentResultsCreated}`);
|
||
console.log(`surfaceElosCreated: ${report.surfaceElosCreated}`);
|
||
if (report.warnings.length) {
|
||
console.log("\nWARNINGS:");
|
||
for (const w of report.warnings) console.log(` ${w}`);
|
||
}
|
||
if (report.errors.length) {
|
||
console.log("\nERRORS:");
|
||
for (const e of report.errors) console.log(` ${e}`);
|
||
process.exit(2);
|
||
}
|
||
}
|
||
|
||
main().catch((e) => {
|
||
console.error(e);
|
||
process.exit(1);
|
||
});
|
||
```
|
||
|
||
- [ ] **Step 2: Add npm script**
|
||
|
||
In `package.json` scripts block, add:
|
||
|
||
```json
|
||
"backfill:canonical": "dotenv -- tsx scripts/backfill-cli.ts"
|
||
```
|
||
|
||
- [ ] **Step 3: Run dry-run locally**
|
||
|
||
Against the dev DB (loaded with `npm run db:sync-prod`):
|
||
|
||
Run: `npm run backfill:canonical -- --dry-run`
|
||
Expected: Non-zero `tournamentsCreated` etc., zero `errors`. Save stdout as `/tmp/backfill-dryrun.txt`.
|
||
|
||
- [ ] **Step 4: Inspect the dry-run output**
|
||
|
||
Review `/tmp/backfill-dryrun.txt`. For each category, sanity-check the count. For example, if we have 2 in-flight golf windows × 4 events each = 8 golf events, `tournamentsCreated` might be 4–8 depending on overlap.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add scripts/backfill-cli.ts package.json
|
||
git commit -m "feat(backfill): CLI wrapper with dry-run default"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: Go/no-go verification (real run on staging)
|
||
|
||
**Human in the loop** — this task is NOT automatable.
|
||
|
||
- [ ] **Step 1: Deploy all preceding commits to staging**
|
||
|
||
Via the project's normal deploy flow.
|
||
|
||
- [ ] **Step 2: Capture a pre-backfill snapshot of the in-flight windows**
|
||
|
||
On staging, run:
|
||
|
||
```bash
|
||
npx tsx scripts/capture-baseline.ts --out=/tmp/pre-backfill-staging
|
||
```
|
||
|
||
- [ ] **Step 3: Run dry-run on staging**
|
||
|
||
```bash
|
||
npm run backfill:canonical -- --dry-run
|
||
```
|
||
|
||
Expected: zero errors; counts match expectations.
|
||
|
||
- [ ] **Step 4: Human review of dry-run report**
|
||
|
||
Requires a human. Verify:
|
||
- Tournament names match real-world identity (no "Masters Tournament 2026" with year=null).
|
||
- Participant counts per sport are sensible.
|
||
- No conflict errors.
|
||
|
||
If anything looks wrong, fix in code (not in DB), redeploy, re-run dry-run.
|
||
|
||
- [ ] **Step 5: Run real backfill on staging**
|
||
|
||
```bash
|
||
npm run backfill:canonical -- --apply
|
||
```
|
||
|
||
Expected: Same counts as dry-run, zero errors.
|
||
|
||
- [ ] **Step 6: Verify no drift on QP totals**
|
||
|
||
```bash
|
||
npx tsx scripts/capture-baseline.ts --out=/tmp/post-backfill-staging
|
||
diff /tmp/pre-backfill-staging/ /tmp/post-backfill-staging/
|
||
```
|
||
|
||
Expected: **Zero differences in `qp-totals-*.json` and `event-results-*.json`**. Backfill must not touch these.
|
||
|
||
The `surface-elos-*.json` files may differ because this captures per-window Elo before Phase 2; after Phase 2 it still captures the same per-window table (`season_participant_surface_elos`), which is untouched by backfill. So diff should still be empty. If not, stop and investigate.
|
||
|
||
- [ ] **Step 7: Spot-check Masters 2026 round-trip on staging**
|
||
|
||
```bash
|
||
psql $STAGING_DB -c "
|
||
SELECT er.placement AS ev_placement, er.raw_score AS ev_score,
|
||
tr.placement AS tr_placement, tr.raw_score AS tr_score,
|
||
sp.name
|
||
FROM event_results er
|
||
JOIN scoring_events se ON se.id = er.scoring_event_id
|
||
JOIN tournaments t ON t.id = se.tournament_id
|
||
JOIN tournament_results tr ON tr.tournament_id = t.id
|
||
AND tr.participant_id = (SELECT participant_id FROM season_participants WHERE id = er.season_participant_id)
|
||
JOIN season_participants sp ON sp.id = er.season_participant_id
|
||
WHERE t.name = 'Masters Tournament' AND t.year = 2026;
|
||
"
|
||
```
|
||
|
||
Expected: Every row has `ev_placement = tr_placement` and `ev_score = tr_score`.
|
||
|
||
- [ ] **Step 8: Proceed to production only if all checks pass**
|
||
|
||
Same flow on prod: capture baseline → dry-run → human review → apply → diff → Masters spot-check.
|
||
|
||
- [ ] **Step 9: Tag**
|
||
|
||
```bash
|
||
git tag phase-2-complete
|
||
```
|
||
|
||
---
|
||
|
||
## Self-Review Checklist
|
||
|
||
- [ ] `runBackfill` with `dryRun: true` writes nothing (verify via row-count diff).
|
||
- [ ] All tests in `scripts/__tests__/backfill-canonical-layer.test.ts` pass.
|
||
- [ ] After real run on prod: every `scoringEvents` row in a `qualifying_points` season has non-null `tournament_id`.
|
||
- [ ] Every `seasonParticipants` row in a `qualifying_points` season has non-null `participant_id`.
|
||
- [ ] Every completed `eventResults` row (non-null `placement` OR non-null `rawScore`) has a matching `tournamentResults` row.
|
||
- [ ] No drift in `participant_qualifying_totals` vs. baseline.
|
||
- [ ] Tag `phase-2-complete` on main.
|
||
|
||
## Rollback
|
||
|
||
Phase 2 is reversible because no production code reads canonical tables yet:
|
||
|
||
```sql
|
||
BEGIN;
|
||
UPDATE season_participants SET participant_id = NULL WHERE participant_id IS NOT NULL;
|
||
UPDATE scoring_events SET tournament_id = NULL WHERE tournament_id IS NOT NULL;
|
||
DELETE FROM tournament_results;
|
||
DELETE FROM participant_surface_elos;
|
||
DELETE FROM participants;
|
||
DELETE FROM tournaments;
|
||
COMMIT;
|
||
```
|
||
|
||
Then re-run backfill after fixing root cause.
|
||
|
||
Phase 3 begins only after `phase-2-complete` has been green in production for 24 hours.
|