IncidentReliability & safetyBy DanPublished Tested Updated
My agents' first game benchmark was wrong. Here's how it was caught
All reports
- ChatGPT + Codex built 33 new DMR_REMOTE toolsField testOct 23
- Gemini 4 Argon reviewed a finished episode: 14 clips, one flag, one inconsistencyField testOct 21
- Does an extra AI review step pay off? A small DeepSeek trial said noHead-to-headOct 19
- My real DeepSeek bill: 12.31 billion tokens for $122.28Cost receiptOct 17
- Gemini 4 Argon kept hanging in Gemini CLI. The cause was a 60-second timeoutIncidentOct 14
- I asked 8 AI coding agents for their biggest weaknessSelf-reportOct 14
- My agents' first game benchmark was wrong. Here's how it was caughtIncidentOct 10
- DeepSeek Flash answered 200 OK and sent nothing for two hoursIncidentSep 14
- DeepSeek V4 Flash audited 18 of my real coding sessions. Two said 'done' too earlyAuditAug 4
The report
Verdict
Invalid: retired and archived
A 3,276-game ranking credited wins by seat instead of by deck, so one deck started every game. An agent review caught it, the ranking was retired, and the rebuilt benchmark was preregistered.
On the tier list: A reliability record, not a ranking. It places no agent.
What went wrong
My unofficial Godforge Duels tools include a simulator that plays community decks against each other. The first ranking, 3,276 games, was built by AI agents. A review found that the runner swapped the deck order and the first player together, so the lower-index deck started every single game, and wins were credited by seat instead of by deck.
How it was caught
Not by luck: by a second agent whose job was to review the first one's work, and an audit that reproduced the review. The ranking was retired the same day and kept in an archive as the record of what was wrong. The benchmark also surfaced one real engine bug, which was fixed.
What replaced it
The test plan was recorded before the rebuilt benchmark ran: 202,400 main games plus 50,600 games to check the result. Both players used the same fixed playing rules. The top deck scored 79.4%, with a 95% confidence interval of 78.8–80.0% showing the statistical uncertainty. This measures a deck under those computer playing rules; a skilled person may play it differently.
Limits of this report
- The source does not say which model wrote the first runner, so this is not a verdict on any one agent.
- A game simulator, not a coding benchmark.
Receipts
DanManREAL · AI lab
Test receipt
AI-built benchmark runner
- Games in the invalid ranking
- 3,276
- Retired
- 2026-10-10
- The bug
- lower-index deck started every gamedeck order and first player were swapped together; wins were credited by seat
- Real engine bug found along the way
- 1, fixed
- Rebuilt benchmark, preregistered
- 202,400 gamesplus 50,600 confirmation games
- Top deck in the rebuilt run
- 79.4%95% CI 78.8–80.0, same pinned policy on both seats
Verdict
Invalid: retired and archived
Tier list A reliability record, not a ranking. It places no agent.
danmanreal.com/tests/agent-built-benchmark-invalid
What was tested
Related
Independent testing. Sponsorships are always disclosed, and verdicts are never for sale.
Disagree? Bring your receipts