chrisp/brackt
1
1
Fork 0
Code Issues 42 Pull requests Projects Releases Packages Wiki Activity Actions 2

Fix flaky tennis simulator test: increase trials and lower threshold

Browse source 
500 trials with a ~3–5% expected win rate had enough variance to
occasionally land below the 0.03 threshold (~2% failure rate).
Bumping to 2000 trials reduces the std dev by 2x; lowering the
threshold to 0.02 keeps the assertion meaningful (still well above
random 0.78%) while eliminating the flakiness.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Chris Parsons 2026-03-25 08:42:01 -07:00
parent 243516263d
commit f77b5fca74
1 changed files with 5 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
app/services/simulations/__tests__/tennis-simulator.test.ts
View file

@ -265,7 +265,7 @@ describe("QP accumulation ranking (unit)", () => {
const eloMap = makeEloMap(IDS, elos); const eloMap = makeEloMap(IDS, elos);
let seed1Wins = 0; let seed1Wins = 0;
const TRIALS = 500; const TRIALS = 2000;
for (let i = 0; i < TRIALS; i++) { for (let i = 0; i < TRIALS; i++) {
const draw = buildDraw(IDS, eloMap); const draw = buildDraw(IDS, eloMap);
const results = simulateMajor(draw, eloMap, "hard"); const results = simulateMajor(draw, eloMap, "hard");
@ -273,9 +273,10 @@ describe("QP accumulation ranking (unit)", () => {
if (seed1?.qp === 20) seed1Wins++; if (seed1?.qp === 20) seed1Wins++;
} }
// Seed 1 has ~3 points advantage per player; against 127 opponents should win // Seed 1 has ~3 Elo points advantage per seed; should win substantially
// substantially more than 1/128 ≈ 0.78% of the time // more than 1/128 ≈ 0.78% of the time. Threshold at 0.02 is still well
// above random; 2000 trials keeps std dev small enough to be non-flaky.
const winRate = seed1Wins / TRIALS; const winRate = seed1Wins / TRIALS;
expect(winRate).toBeGreaterThan(0.03); // > 3% (well above random 0.78%) expect(winRate).toBeGreaterThan(0.02); // > 2% (well above random 0.78%)
}); });
}); });