brackt/scripts/backfill-canonical-layer.ts
Chris Parsons 2848231235
Canonical tournament layer: schema + backfill (1/2) (#365)
* refactor(schema): rename per-window tables to season_* prefix

Renames participants, participant_expected_values, participant_qualifying_totals,
participant_results, participant_surface_elos to season_* prefixed names.
Renames event_results.participant_id to season_participant_id.
Phase 1a of canonical tournament layer migration.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor: rename participant.ts model file to season-participant.ts

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(models): update model layer to use renamed schema exports

Updated all model files to use the renamed schema exports from Task 1:
- participants → seasonParticipants
- participantExpectedValues → seasonParticipantExpectedValues
- participantQualifyingTotals → seasonParticipantQualifyingTotals
- participantResults → seasonParticipantResults
- participantSurfaceElos → seasonParticipantSurfaceElos
- eventResults.participantId → eventResults.seasonParticipantId
- db.query relation accessors updated
- Relation field .participant → .seasonParticipant where applicable
- Import paths updated: ./participant → ./season-participant

Files updated (14 model files + 3 test files):
- draft-pick.ts
- draft-utils.ts
- event-result.ts
- group-stage-match.ts
- participant-result.ts
- qualifying-points.ts
- scoring-calculator.ts
- scoring-event.ts
- sports-season.ts
- surface-elo.ts
- team-score-events.ts
- cs2-major-stage.ts
- golf-skills.ts
- participant-expected-value.ts
- __tests__/sports-season.clone.test.ts
- __tests__/auto-pick.test.ts
- __tests__/executeAutoPick.timer.test.ts

Typecheck errors decreased: 779 → 499 (280 fewer)
All model file errors related to renamed schemas resolved.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(routes): update route layer to use renamed schema exports

- Update model import from ~/models/participant to ~/models/season-participant
- Rename schema.participants to schema.seasonParticipants
- Rename schema.participantResults to schema.seasonParticipantResults
- Rename db.query.participants to db.query.seasonParticipants
- Update 9 route files and 1 test file

Affected files:
- admin.sports-seasons.$id.events.$eventId.bracket.server.ts
- admin.sports-seasons.$id.participants.tsx
- api/draft.force-manual-pick.ts
- api/draft.make-pick.ts
- api/draft.replace-pick.ts
- api/seasons.$seasonId.draft.ts
- leagues/$leagueId.draft-board.$seasonId.tsx
- leagues/$leagueId.sports-seasons.$sportsSeasonId.server.ts
- admin/__tests__/sports-seasons-participants.test.ts

Error count reduced from 499 to 453 (46 errors fixed).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(routes): update route files for schema rename

Update route imports from ~/models/participant to ~/models/season-participant
and fix references to .participant/.participantId on event results to use
.seasonParticipant/.seasonParticipantId after schema rename.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* refactor(services): update simulators and services for renamed schema

Update all simulators, services, and server files to use renamed schema tables:
- participants → seasonParticipants
- participantExpectedValues → seasonParticipantExpectedValues
- participantResults → seasonParticipantResults
- eventResults.participantId → eventResults.seasonParticipantId

Files updated:
- 20 sport simulators (NBA, NHL, NFL, MLB, etc.)
- probability-updater.ts
- standings-sync/index.ts
- sports-data-sync.server.ts
- server/socket.ts

Typecheck errors reduced from 365 to 0.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* migration: rename per-window tables to season_* prefix

* fix(tests): update mock query keys after participants table rename

Change mock db.query.participants to db.query.seasonParticipants in test
files to match the schema rename from commit 66145a9. This fixes
"Cannot read properties of undefined (reading 'findFirst'/'findMany')"
errors that occurred when production code queries db.query.seasonParticipants
but test mocks only defined the old participants key.

Files updated:
- app/services/simulations/__tests__/world-cup-simulator.test.ts
- app/routes/api/__tests__/draft.force-manual-pick.test.ts
- app/routes/api/__tests__/draft.force-manual-pick.timer-mode.test.ts
- app/routes/api/__tests__/draft.make-pick.timer-mode.test.ts
- server/__tests__/timer-autodraft.test.ts
- app/models/__tests__/team-score-events.test.ts

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(tests): update remaining mock paths and keys after schema rename

* fix(tests): final two mock stragglers after schema rename

- draft-pick.test.ts: assertion on db.query.participantQualifyingTotals
- process-match-result.test.ts: mock key participants → seasonParticipants

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore: add post-phase1a baseline capture (temp, for diff verification)

* chore: capture pre-migration baselines

* chore: remove post-phase1a capture helper after verification

* schema: add canonical tournament & participant tables

Adds tournaments, participants (canonical), tournament_results, and
participant_surface_elos (canonical). Adds nullable tournament_id to
scoring_events and nullable participant_id to season_participants.
Phase 1b of canonical tournament layer migration.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(models): add canonical tournament, participant, result, surface-elo models

Adds CRUD modules for the canonical tables created in commit 775b905.
Each module mirrors existing app/models conventions (database() from
~/database/context, schema from ~/database/schema, mock-based tests).

Key implementation notes:
- participant.ts exports use "Canonical" prefix (CanonicalParticipant,
  createCanonicalParticipant, etc.) to avoid collision with existing
  season-participant.ts exports
- All four models include comprehensive unit tests following the
  audit-log.test.ts pattern
- Tests use mocked db responses (no real database access)
- Upsert functions use onConflictDoUpdate for appropriate unique constraints

Part of Phase 1b of canonical tournament layer migration.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* migration: create canonical tables, add nullable FKs

* scripts: add extractTournamentIdentity helper for backfill

Pure function that derives canonical (name, year) identity from a
scoring_events row, stripping trailing 4-digit years from the name or
falling back to eventDate. Used by the Phase 2 backfill to group
per-window events into canonical tournaments.

* scripts: add backfill orchestrator for canonical layer

Populates canonical tournaments, participants, tournament_results, and
participant_surface_elos from per-window data for qualifying-points
sports. Skips already-linked rows, is idempotent, and supports dry-run
mode.

Critical invariants enforced by the implementation:
- qualifying_points_awarded is never copied to tournament_results
- season_participant_qualifying_totals is never touched
- conflicting surface-Elo values between windows raise a loud error
  (recorded in report.errors) rather than overwriting

* scripts: add backfill CLI with dry-run default

Wires backfill-canonical-layer.ts to a CLI entry point exposed as
`npm run backfill:canonical`. Defaults to --dry-run; requires --apply
to actually write. Supports --sport=<uuid> to limit to a single sport.
Exits 2 if the backfill reports errors (e.g., surface-Elo conflicts).

* fix(backfill-cli): wrap runBackfill in DatabaseContext.run

The orchestrator uses database() from ~/database/context, which requires
AsyncLocalStorage to be populated. Wrap the CLI invocation with
DatabaseContext.run(db, ...) using server/db's cached connection pool.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(backfill-cli): exit 0 on success so pg pool doesn't block

The cached postgres connection pool keeps the Node event loop open after
main() returns. Explicit process.exit(0) on success mirrors the pattern
in scripts/capture-baseline.ts.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Chris Parsons <chrisp@extrahop.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-01 20:13:18 -07:00

358 lines
11 KiB
TypeScript

/**
* Phase 2 one-off backfill: populate canonical tables
* (`tournaments`, `participants`, `tournament_results`,
* `participant_surface_elos`) from existing per-window data.
*
* See CLAUDE.md and the Phase 2 plan docs. Rules that this script
* MUST obey:
* - Never copy `qualifying_points_awarded` from `event_results` to
* `tournament_results`. QP stays per-window.
* - Never touch `season_participant_qualifying_totals`.
* - Abort loud (collect into `report.errors`) if two windows disagree
* on a canonical participant's surface-Elo values.
*
* `dryRun: true` means: run every read, compute every count, but never
* issue an `insert()` or `update()`.
*/
import { eq, and, isNull } from "drizzle-orm";
import { database } from "~/database/context";
import * as schema from "~/database/schema";
import { extractTournamentIdentity } from "./backfill/match-tournament";
export interface BackfillOptions {
dryRun: boolean;
sportId?: string;
}
export interface BackfillReport {
tournamentsCreated: number;
/** Count of scoring_events whose tournament_id was (or would be) set. */
tournamentsLinked: number;
participantsCreated: number;
/** Count of season_participants whose participant_id was (or would be) set. */
participantsLinked: number;
tournamentResultsCreated: number;
surfaceElosCreated: number;
warnings: string[];
errors: string[];
}
type Db = ReturnType<typeof database>;
type SportsSeasonRow = typeof schema.sportsSeasons.$inferSelect;
type ScoringEventRow = typeof schema.scoringEvents.$inferSelect;
type SeasonParticipantRow = typeof schema.seasonParticipants.$inferSelect;
type EventResultRow = typeof schema.eventResults.$inferSelect;
type SeasonParticipantSurfaceEloRow =
typeof schema.seasonParticipantSurfaceElos.$inferSelect;
type TournamentRow = typeof schema.tournaments.$inferSelect;
type ParticipantRow = typeof schema.participants.$inferSelect;
type TournamentResultRow = typeof schema.tournamentResults.$inferSelect;
type ParticipantSurfaceEloRow =
typeof schema.participantSurfaceElos.$inferSelect;
export async function runBackfill(
opts: BackfillOptions,
): Promise<BackfillReport> {
const report: BackfillReport = {
tournamentsCreated: 0,
tournamentsLinked: 0,
participantsCreated: 0,
participantsLinked: 0,
tournamentResultsCreated: 0,
surfaceElosCreated: 0,
warnings: [],
errors: [],
};
const db = database();
// 1. Load qualifying-points seasons (optionally filtered by sport).
const whereClauses = [
eq(schema.sportsSeasons.scoringPattern, "qualifying_points"),
];
if (opts.sportId) {
whereClauses.push(eq(schema.sportsSeasons.sportId, opts.sportId));
}
const seasons = (await db
.select()
.from(schema.sportsSeasons)
.where(and(...whereClauses))) as SportsSeasonRow[];
for (const season of seasons) {
await backfillSeason(db, season, opts, report);
}
return report;
}
async function backfillSeason(
db: Db,
season: SportsSeasonRow,
opts: BackfillOptions,
report: BackfillReport,
): Promise<void> {
// ─── a. Tournament linking ────────────────────────────────────────────────
const unlinkedEvents = (await db
.select()
.from(schema.scoringEvents)
.where(
and(
eq(schema.scoringEvents.sportsSeasonId, season.id),
isNull(schema.scoringEvents.tournamentId),
),
)) as ScoringEventRow[];
for (const ev of unlinkedEvents) {
let identity;
try {
identity = extractTournamentIdentity({
name: ev.name,
eventDate: ev.eventDate,
eventType: ev.eventType,
});
} catch (e) {
report.warnings.push(
`skip scoring_event ${ev.id} (${ev.name}): ${(e as Error).message}`,
);
continue;
}
// Look up existing canonical tournament.
const [existing] = (await db
.select()
.from(schema.tournaments)
.where(
and(
eq(schema.tournaments.sportId, season.sportId),
eq(schema.tournaments.name, identity.name),
eq(schema.tournaments.year, identity.year),
),
)
.limit(1)) as TournamentRow[];
let tournamentId: string | undefined = existing?.id;
if (!existing) {
const startsAt = ev.eventDate ? new Date(ev.eventDate) : null;
const status =
startsAt && startsAt.getTime() < Date.now() ? "completed" : "scheduled";
if (!opts.dryRun) {
const [inserted] = (await db
.insert(schema.tournaments)
.values({
sportId: season.sportId,
name: identity.name,
year: identity.year,
startsAt,
status,
})
.returning()) as TournamentRow[];
tournamentId = inserted.id;
}
report.tournamentsCreated += 1;
}
if (!opts.dryRun && tournamentId) {
await db
.update(schema.scoringEvents)
.set({ tournamentId, updatedAt: new Date() })
.where(eq(schema.scoringEvents.id, ev.id));
}
report.tournamentsLinked += 1;
}
// ─── b. Participant linking ───────────────────────────────────────────────
const unlinkedParticipants = (await db
.select()
.from(schema.seasonParticipants)
.where(
and(
eq(schema.seasonParticipants.sportsSeasonId, season.id),
isNull(schema.seasonParticipants.participantId),
),
)) as SeasonParticipantRow[];
for (const sp of unlinkedParticipants) {
const [existing] = (await db
.select()
.from(schema.participants)
.where(
and(
eq(schema.participants.sportId, season.sportId),
eq(schema.participants.name, sp.name),
),
)
.limit(1)) as ParticipantRow[];
let participantId: string | undefined = existing?.id;
if (!existing) {
if (!opts.dryRun) {
const [inserted] = (await db
.insert(schema.participants)
.values({
sportId: season.sportId,
name: sp.name,
})
.returning()) as ParticipantRow[];
participantId = inserted.id;
}
report.participantsCreated += 1;
}
if (!opts.dryRun && participantId) {
await db
.update(schema.seasonParticipants)
.set({ participantId, updatedAt: new Date() })
.where(eq(schema.seasonParticipants.id, sp.id));
}
report.participantsLinked += 1;
}
// ─── c. Tournament results (copy completed event_results) ────────────────
// Refetch events — they may now have tournamentId set (if !dryRun).
const allEvents = (await db
.select()
.from(schema.scoringEvents)
.where(
eq(schema.scoringEvents.sportsSeasonId, season.id),
)) as ScoringEventRow[];
for (const ev of allEvents) {
if (!ev.tournamentId) {
// In dry-run we may not have a tournamentId yet; skip result copy.
continue;
}
const results = (await db
.select()
.from(schema.eventResults)
.where(
eq(schema.eventResults.scoringEventId, ev.id),
)) as EventResultRow[];
for (const r of results) {
// Only copy rows with real placement/rawScore data.
if (r.placement == null && r.rawScore == null) {
continue;
}
// Look up the season_participants row to get canonical participantId.
const [sp] = (await db
.select()
.from(schema.seasonParticipants)
.where(
eq(schema.seasonParticipants.id, r.seasonParticipantId),
)
.limit(1)) as SeasonParticipantRow[];
if (!sp || !sp.participantId) {
// Link step should have handled this; skip defensively.
continue;
}
// Check if a tournament_results row already exists.
const [existingResult] = (await db
.select()
.from(schema.tournamentResults)
.where(
and(
eq(schema.tournamentResults.tournamentId, ev.tournamentId),
eq(schema.tournamentResults.participantId, sp.participantId),
),
)
.limit(1)) as TournamentResultRow[];
if (existingResult) {
continue;
}
if (!opts.dryRun) {
// NOTE: intentionally do NOT copy qualifyingPointsAwarded.
await db.insert(schema.tournamentResults).values({
tournamentId: ev.tournamentId,
participantId: sp.participantId,
placement: r.placement,
rawScore: r.rawScore,
});
}
report.tournamentResultsCreated += 1;
}
}
// ─── d. Surface Elo (per-window → canonical) ─────────────────────────────
const elos = (await db
.select()
.from(schema.seasonParticipantSurfaceElos)
.where(
eq(schema.seasonParticipantSurfaceElos.sportsSeasonId, season.id),
)) as SeasonParticipantSurfaceEloRow[];
for (const elo of elos) {
const [sp] = (await db
.select()
.from(schema.seasonParticipants)
.where(
eq(schema.seasonParticipants.id, elo.participantId),
)
.limit(1)) as SeasonParticipantRow[];
if (!sp || !sp.participantId) {
continue;
}
const canonicalParticipantId = sp.participantId;
const [existingElo] = (await db
.select()
.from(schema.participantSurfaceElos)
.where(
eq(
schema.participantSurfaceElos.participantId,
canonicalParticipantId,
),
)
.limit(1)) as ParticipantSurfaceEloRow[];
if (existingElo) {
// Conflict detection: compare eloHard/eloClay/eloGrass/worldRanking.
const fields: Array<keyof ParticipantSurfaceEloRow> = [
"eloHard",
"eloClay",
"eloGrass",
"worldRanking",
];
const mismatches: string[] = [];
for (const f of fields) {
if (existingElo[f] !== elo[f as keyof SeasonParticipantSurfaceEloRow]) {
mismatches.push(
`${f}: existing=${String(existingElo[f])} vs incoming=${String(elo[f as keyof SeasonParticipantSurfaceEloRow])}`,
);
}
}
if (mismatches.length > 0) {
report.errors.push(
`conflict for participant ${canonicalParticipantId} (season ${season.id}): ${mismatches.join(", ")}`,
);
}
// Do not overwrite.
continue;
}
if (!opts.dryRun) {
await db.insert(schema.participantSurfaceElos).values({
participantId: canonicalParticipantId,
eloHard: elo.eloHard,
eloClay: elo.eloClay,
eloGrass: elo.eloGrass,
worldRanking: elo.worldRanking,
});
}
report.surfaceElosCreated += 1;
}
}