brackt/docs/agents/simulators.md

171 lines
9.4 KiB
Markdown
Raw Normal View History

Formalize simulator system with manifest, input-policy, runner, and admin UI (#409) Introduces three new schema tables (simulator_profiles, sports_season_simulator_configs, season_participant_simulator_inputs), a central model layer (app/models/simulator.ts), and a single runner entry point so every simulator run follows the same prepare → simulate → persist → snapshot → recalculate flow. Key additions: - manifest.ts: per-simulator display names, default configs, required/ optional inputs, derivable-input declarations, and setup sections - input-policy.ts: resolves sourceElo from projectedWins, projectedTablePoints, or sourceOdds; resolves ratings from sourceOdds; supports block / fallbackElo / averageKnown / worstKnownMinus strategies - runner.ts: single entry point for admin simulation runs; materialises derived inputs, normalises result columns, zeroes omitted participants, snapshots EVs, and recalculates linked fantasy standings - /admin/simulators: inventory page with per-season readiness and bulk run - /admin/sports-seasons/:id/simulator: per-season setup page with readiness summary, input-policy editor, raw JSON config override, and CSV bulk input - NCAAM/NCAAW simulators now read ratings from season_participant_simulator_inputs, falling back to the hardcoded name-keyed maps while DB data is being populated - Clone flow copies simulator config by default; volatile inputs (odds, Elo) only copied when explicitly requested Code-review fixes included in this commit: - source field in compatibility bridge checked with !== null instead of !== undefined - sourceEloRequirementLabel no longer appends "configured fallback" when the participant is already excluded from all resolved sources - Duplicate inline label maps in input-policy.ts replaced with simulatorInputLabel - save-config preserves existing inputPolicy when the submitted JSON omits it - Input table truncation label added (Showing 20 of N) - CSV description notes values must not contain commas - N+1 comment added to listSportsSeasonSimulatorSummaries - assertRegistrySchemaDriftFree called in manifest tests - Runner test suite added covering happy path, already-running guard, readiness failure, empty results, and error recovery with status reset Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 21:09:53 -07:00
# Simulator Guide
Brackt EV simulators project real-world sports outcomes into the standard top-eight probability shape used by fantasy expected values.
## Architecture
- `app/services/simulations/registry.ts` maps each `simulatorType` to the algorithm implementation.
- `app/services/simulations/manifest.ts` describes each simulator for admin/UI use: display name, default config, required inputs, optional inputs, and setup sections.
- `app/services/simulations/input-policy.ts` resolves acceptable alternate inputs, such as projected wins or futures odds, into simulator-ready Elo values.
- `app/models/simulator.ts` owns simulator profile/config/input queries. Routes should use this model layer and must not query Drizzle directly.
- `app/services/simulations/runner.ts` is the only supported way for admin routes to run a sports-season simulation.
- `season_participant_expected_values` is an output table. New mutable simulator inputs belong in `season_participant_simulator_inputs`.
## Data Flow
1. Admin configures the sport's default `simulatorType` on the sport.
2. Each `sports_seasons` row gets its own simulator config through `sports_season_simulator_configs`.
3. Admin/import flows save participant inputs to `season_participant_simulator_inputs`.
4. The runner validates readiness, runs the registered simulator, normalizes output columns, persists EVs, snapshots probabilities, zeroes omitted participants, and recalculates linked fantasy standings.
5. `/admin/simulators` inventories all sports-season simulators and can run ready/idle seasons.
Before running older Elo-based simulators, the runner materializes resolved Elo values back through `season_participant_simulator_inputs` and the legacy EV compatibility bridge. This lets older simulator algorithms keep reading `sourceElo` while admin setup can use direct Elo, projected wins/table points, futures odds, or an explicit fallback policy.
The runner also materializes derived `rating` values for rating-based simulators that declare that support, such as preseason NCAAM/NCAAW using futures odds before KenPom/Barttorvik-style ratings are available.
## Preseason Draftability
Brackt sports are expected to be draftable before their real-world season starts. When adding or revising a team-sport simulator, design for preseason and in-season runs first:
- Include all draftable teams/participants, not only the eventual playoff field.
- Simulate regular-season qualification or standings when playoff participation is not known yet.
- Let current standings override or blend with preseason projections as the season progresses.
- Use projected wins, projected table points, futures odds, direct ratings, or sport-specific ratings as season-scoped inputs that can drive preseason EVs.
- Switch to bracket-aware simulation only when a real bracket or finalized seeds exist.
A playoff-only simulator is not sufficient for team sports that Brackt drafts in preseason. If a sport ever has a true exception, that exception must be explicit in the sport-specific implementation plan and product requirements.
Formalize simulator system with manifest, input-policy, runner, and admin UI (#409) Introduces three new schema tables (simulator_profiles, sports_season_simulator_configs, season_participant_simulator_inputs), a central model layer (app/models/simulator.ts), and a single runner entry point so every simulator run follows the same prepare → simulate → persist → snapshot → recalculate flow. Key additions: - manifest.ts: per-simulator display names, default configs, required/ optional inputs, derivable-input declarations, and setup sections - input-policy.ts: resolves sourceElo from projectedWins, projectedTablePoints, or sourceOdds; resolves ratings from sourceOdds; supports block / fallbackElo / averageKnown / worstKnownMinus strategies - runner.ts: single entry point for admin simulation runs; materialises derived inputs, normalises result columns, zeroes omitted participants, snapshots EVs, and recalculates linked fantasy standings - /admin/simulators: inventory page with per-season readiness and bulk run - /admin/sports-seasons/:id/simulator: per-season setup page with readiness summary, input-policy editor, raw JSON config override, and CSV bulk input - NCAAM/NCAAW simulators now read ratings from season_participant_simulator_inputs, falling back to the hardcoded name-keyed maps while DB data is being populated - Clone flow copies simulator config by default; volatile inputs (odds, Elo) only copied when explicitly requested Code-review fixes included in this commit: - source field in compatibility bridge checked with !== null instead of !== undefined - sourceEloRequirementLabel no longer appends "configured fallback" when the participant is already excluded from all resolved sources - Duplicate inline label maps in input-policy.ts replaced with simulatorInputLabel - save-config preserves existing inputPolicy when the submitted JSON omits it - Input table truncation label added (Showing 20 of N) - CSV description notes values must not contain commas - N+1 comment added to listSportsSeasonSimulatorSummaries - assertRegistrySchemaDriftFree called in manifest tests - Runner test suite added covering happy path, already-running guard, readiness failure, empty results, and error recovery with status reset Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 21:09:53 -07:00
## Season Scoping
Never store mutable simulator state on `sports`. A sport can have multiple active seasons at once, such as NHL playoffs for one season and preseason drafting for the next. Each sports season needs independent:
- Participants
- Simulator config overrides
- Ratings, rankings, odds, seeds, and metadata
- Standings/events/brackets
- EV outputs and snapshots
When cloning a sports season, copy simulator structure/config by default. Do not copy volatile inputs such as odds, Elo ratings, rankings, or standings unless an admin explicitly opts in.
## Adding A Simulator
1. Add the simulator type to `database/schema.ts` and generate a Drizzle migration with `npm run db:generate`.
2. Implement `Simulator` in `app/services/simulations/`.
3. Register the simulator in `app/services/simulations/registry.ts`.
4. Add a manifest profile in `app/services/simulations/manifest.ts`.
5. Add readiness and setup expectations through required/optional inputs and setup sections.
6. If a required input can be derived, add `derivableInputs`. For example, an Elo simulator with projected wins support should declare `derivableInputs: { sourceElo: ["projectedWins"] }`.
7. Add unit tests for algorithm helpers and a readiness/manifest regression test.
8. Confirm `/admin/simulators` shows the season and the per-season setup page reports actionable missing inputs.
## Hardcoding Rules
Do not hardcode refreshable season data in simulator files:
- No current-season Elo tables
- No KenPom/Barttorvik/SRS maps as production inputs
- No bookmaker odds
- No team rankings/seeds that change per season
- No participant-specific metadata that admins may need to change
It is acceptable to keep stable algorithmic rules in code:
- Bracket advancement mechanics
- Swiss-stage mechanics
- Best-of series math
- Plackett-Luce, Elo, Log5, and Harville formulas
- Sport format rules that are intentionally not admin-editable yet
If a hardcoded fallback is temporarily retained for backwards compatibility, admin readiness should still require the corresponding DB-managed input before normal admin-triggered runs.
## Admin Setup
Use `/admin/sports-seasons/:id/simulator` for per-season setup. The page should show readiness, missing inputs, config JSON, links to sport-specific setup pages, and a generic CSV importer for common inputs:
```csv
name,sourceElo,sourceOdds,worldRanking,rating,projectedWins,projectedTablePoints,seed,region
Boston Celtics,1699,+450,1,0.95,58,,1,East
```
Keep specialized pages when they provide real workflow value, such as Golf Skills, Surface Elo, CS2 setup, regular standings, and bracket setup. Those pages should still feed readiness through the simulator model layer.
## Input Policies
Make MLB projected wins actually drive the simulation Entering projected wins for an in-progress MLB season did not behave as expected: the entered numbers came back changed, and the simulation appeared to ignore them in favour of whatever Elo was already stored. Four separate defects were involved. Projections are now stored and shown verbatim. The Elo Ratings page never kept the number typed into it — the field was a display derived from Elo, so a pasted 95 rendered as 95.1 the moment it was applied (wins to Elo rounds to an integer Elo) and drifted again after each run, because a run re-resolves that Elo through the input policy. The loader now reads back the stored projection and the paste flow keeps the pasted value as-is; the derived round-trip survives only as a prefill for seasons that have never had a projection saved. A stale Elo no longer silently outranks a projection. baseEloPriority takes the first available base source, and the simulator page's bulk CSV wrote projectedWins without stamping metadata.sourceEloMethod, so the non-destructive upsert left the old Elo in place as a trusted direct value and it won the race — the projection was stored and then ignored on every run. The CSV path now stamps the flag like the Elo Ratings page does, the metadata upsert merges rather than replaces so a flag-only write keeps unrelated keys, and Base Elo Source is editable per season for the case where a genuine hand-entered Elo should still lose to projections. Projected wins now act as a projected final total. The value was baked into a flat season-long rate (projectedWins / 162) applied to every remaining game, so a team at 60-50 projected for 95 finished around 90.5 and the projection was never reached mid-season. seedingWinRateFor spreads the difference over the games still to play, which is a no-op pre-season where the two rates coincide; projectedWinsWeight blends it back toward the Elo-implied rate. Playoff-parity compression is restored for Elo-rated teams. eloToRDif scaled by RDIF_DIVISOR, making it the exact algebraic inverse of winRateFromRDif, so any team with an Elo skipped the compression every hardcoded-rdif team gets: a 95-win projection became RDif +686 and played playoff games at .586 instead of the documented ~.517. It now scales by SEEDING_RDIF_SCALE, landing at ~+140 alongside the Dodgers' hardcoded +137. Also fixes the preview table's "missing a required input" marker, which flagged every projection-configured participant because a generated Elo or rating is deliberately hidden from getParticipantSimulatorInputs. It now consults the resolved values, so it agrees with readiness. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CQSEmmojmqmGdJttgzqCWK
2026-08-29 05:31:15 +00:00
Direct ratings are preferred by default. If a simulator declares derived inputs, readiness may also pass with those alternatives:
Formalize simulator system with manifest, input-policy, runner, and admin UI (#409) Introduces three new schema tables (simulator_profiles, sports_season_simulator_configs, season_participant_simulator_inputs), a central model layer (app/models/simulator.ts), and a single runner entry point so every simulator run follows the same prepare → simulate → persist → snapshot → recalculate flow. Key additions: - manifest.ts: per-simulator display names, default configs, required/ optional inputs, derivable-input declarations, and setup sections - input-policy.ts: resolves sourceElo from projectedWins, projectedTablePoints, or sourceOdds; resolves ratings from sourceOdds; supports block / fallbackElo / averageKnown / worstKnownMinus strategies - runner.ts: single entry point for admin simulation runs; materialises derived inputs, normalises result columns, zeroes omitted participants, snapshots EVs, and recalculates linked fantasy standings - /admin/simulators: inventory page with per-season readiness and bulk run - /admin/sports-seasons/:id/simulator: per-season setup page with readiness summary, input-policy editor, raw JSON config override, and CSV bulk input - NCAAM/NCAAW simulators now read ratings from season_participant_simulator_inputs, falling back to the hardcoded name-keyed maps while DB data is being populated - Clone flow copies simulator config by default; volatile inputs (odds, Elo) only copied when explicitly requested Code-review fixes included in this commit: - source field in compatibility bridge checked with !== null instead of !== undefined - sourceEloRequirementLabel no longer appends "configured fallback" when the participant is already excluded from all resolved sources - Duplicate inline label maps in input-policy.ts replaced with simulatorInputLabel - save-config preserves existing inputPolicy when the submitted JSON omits it - Input table truncation label added (Showing 20 of N) - CSV description notes values must not contain commas - N+1 comment added to listSportsSeasonSimulatorSummaries - assertRegistrySchemaDriftFree called in manifest tests - Runner test suite added covering happy path, already-running guard, readiness failure, empty results, and error recovery with status reset Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 21:09:53 -07:00
- `projectedWins` can become Elo using `seasonGames` and `parityFactor` from season config.
- `projectedTablePoints` can become Elo using `seasonGames`, `maxTablePoints`, and `parityFactor`.
- `sourceOdds` can become Elo through the shared futures-to-Elo conversion.
- `sourceOdds` can become a generic `rating` when the simulator declares `derivableInputs: { rating: ["sourceOdds"] }`.
Make MLB projected wins actually drive the simulation Entering projected wins for an in-progress MLB season did not behave as expected: the entered numbers came back changed, and the simulation appeared to ignore them in favour of whatever Elo was already stored. Four separate defects were involved. Projections are now stored and shown verbatim. The Elo Ratings page never kept the number typed into it — the field was a display derived from Elo, so a pasted 95 rendered as 95.1 the moment it was applied (wins to Elo rounds to an integer Elo) and drifted again after each run, because a run re-resolves that Elo through the input policy. The loader now reads back the stored projection and the paste flow keeps the pasted value as-is; the derived round-trip survives only as a prefill for seasons that have never had a projection saved. A stale Elo no longer silently outranks a projection. baseEloPriority takes the first available base source, and the simulator page's bulk CSV wrote projectedWins without stamping metadata.sourceEloMethod, so the non-destructive upsert left the old Elo in place as a trusted direct value and it won the race — the projection was stored and then ignored on every run. The CSV path now stamps the flag like the Elo Ratings page does, the metadata upsert merges rather than replaces so a flag-only write keeps unrelated keys, and Base Elo Source is editable per season for the case where a genuine hand-entered Elo should still lose to projections. Projected wins now act as a projected final total. The value was baked into a flat season-long rate (projectedWins / 162) applied to every remaining game, so a team at 60-50 projected for 95 finished around 90.5 and the projection was never reached mid-season. seedingWinRateFor spreads the difference over the games still to play, which is a no-op pre-season where the two rates coincide; projectedWinsWeight blends it back toward the Elo-implied rate. Playoff-parity compression is restored for Elo-rated teams. eloToRDif scaled by RDIF_DIVISOR, making it the exact algebraic inverse of winRateFromRDif, so any team with an Elo skipped the compression every hardcoded-rdif team gets: a 95-win projection became RDif +686 and played playoff games at .586 instead of the documented ~.517. It now scales by SEEDING_RDIF_SCALE, landing at ~+140 alongside the Dodgers' hardcoded +137. Also fixes the preview table's "missing a required input" marker, which flagged every projection-configured participant because a generated Elo or rating is deliberately hidden from getParticipantSimulatorInputs. It now consults the resolved values, so it agrees with readiness. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CQSEmmojmqmGdJttgzqCWK
2026-08-29 05:31:15 +00:00
### Raw Elo vs. projections
Raw Elo and projections are *substitutes*, not a blend: `inputPolicy.baseEloPriority`
lists them in order and the first source a participant has wins outright. The
default is `["sourceElo", "projectedWins", "projectedTablePoints"]`, so a stored Elo
beats a projection. Set the Base Elo Source control on the simulator page (or
`baseEloPriority` directly) to `["projectedWins", "sourceElo"]` when projections are
the season's source of truth. Futures odds are separate — they blend on top of
whichever base won, weighted by `inputPolicy.oddsWeight`.
Whenever you write a projection without an explicit Elo, stamp
`metadata.sourceEloMethod` (`"projectedWins"` / `"projectedTablePoints"`) on the
row. `getParticipantSimulatorInputs` reads that flag and returns `sourceElo: null`
so the Elo is re-derived from the projection on every run. Skip it and the
non-destructive upsert leaves the previous Elo in place as a *direct* value, which
then wins the priority race — the projection is stored and silently ignored. Both
the Elo Ratings page's projections mode and the simulator page's CSV importer do
this; any new importer must too.
Projections are stored and displayed exactly as entered. Never round-trip one
through its derived Elo for display: the conversion rounds to an integer Elo, and a
run re-resolves that Elo through the input policy (clamping, plus any futures
blend), so the number the admin sees drifts away from the number they typed.
### Mid-season projections
A projected win total is a projected *final* total. A simulator that seeds from
projections mid-season must spread the difference over the games still to play —
`(projectedWins - currentWins) / remainingGames` — rather than reusing the
season-long rate the derived Elo encodes, or it will never reach the projection.
See `seedingWinRateFor` in `mlb-simulator.ts` (config knob `projectedWinsWeight`,
1 = the projection is authoritative) and `simulateRegularSeasonSeeds` in
`nll-simulator.ts` (which additionally decays a preseason prior as the season
completes).
Formalize simulator system with manifest, input-policy, runner, and admin UI (#409) Introduces three new schema tables (simulator_profiles, sports_season_simulator_configs, season_participant_simulator_inputs), a central model layer (app/models/simulator.ts), and a single runner entry point so every simulator run follows the same prepare → simulate → persist → snapshot → recalculate flow. Key additions: - manifest.ts: per-simulator display names, default configs, required/ optional inputs, derivable-input declarations, and setup sections - input-policy.ts: resolves sourceElo from projectedWins, projectedTablePoints, or sourceOdds; resolves ratings from sourceOdds; supports block / fallbackElo / averageKnown / worstKnownMinus strategies - runner.ts: single entry point for admin simulation runs; materialises derived inputs, normalises result columns, zeroes omitted participants, snapshots EVs, and recalculates linked fantasy standings - /admin/simulators: inventory page with per-season readiness and bulk run - /admin/sports-seasons/:id/simulator: per-season setup page with readiness summary, input-policy editor, raw JSON config override, and CSV bulk input - NCAAM/NCAAW simulators now read ratings from season_participant_simulator_inputs, falling back to the hardcoded name-keyed maps while DB data is being populated - Clone flow copies simulator config by default; volatile inputs (odds, Elo) only copied when explicitly requested Code-review fixes included in this commit: - source field in compatibility bridge checked with !== null instead of !== undefined - sourceEloRequirementLabel no longer appends "configured fallback" when the participant is already excluded from all resolved sources - Duplicate inline label maps in input-policy.ts replaced with simulatorInputLabel - save-config preserves existing inputPolicy when the submitted JSON omits it - Input table truncation label added (Showing 20 of N) - CSV description notes values must not contain commas - N+1 comment added to listSportsSeasonSimulatorSummaries - assertRegistrySchemaDriftFree called in manifest tests - Runner test suite added covering happy path, already-running guard, readiness failure, empty results, and error recovery with status reset Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 21:09:53 -07:00
Missing tail participants must remain blocked unless the season config explicitly chooses an `inputPolicy.missingEloStrategy`:
```json
{
"inputPolicy": {
"missingEloStrategy": "worstKnownMinus",
"fallbackElo": 1400,
"fallbackEloDelta": 25,
"eloMin": 1100,
"eloMax": 1900
}
}
```
Supported strategies are `block`, `fallbackElo`, `averageKnown`, and `worstKnownMinus`. Use fallbacks only when the tail participants have genuinely tiny EV impact; the setup page will warn when derived or fallback Elo is being used.
For rating conversions, configure `inputPolicy.ratingMin` and `inputPolicy.ratingMax` to match the target model scale. NCAAM uses a KenPom-like net rating range; NCAAW uses a Barthag-like 0-1 range. Futures-derived ratings are a preseason approximation and should be replaced by real ratings when available.
## Required Checks
After simulator changes, run:
```bash
npm run typecheck
npm run lint
npm run test:run
```
At minimum, add or update tests for:
- Manifest/schema/registry drift
- Readiness validation
- Input persistence and legacy compatibility bridge
- Shared runner success/failure behavior
- Any simulator whose required inputs changed