AuditReliability & safetyBy DanPublished Tested Updated
DeepSeek V4 Flash audited 18 of my real coding sessions. Two said 'done' too early
All reports
- ChatGPT + Codex built 33 new DMR_REMOTE toolsField testOct 23
- Gemini 4 Argon reviewed a finished episode: 14 clips, one flag, one inconsistencyField testOct 21
- Does an extra AI review step pay off? A small DeepSeek trial said noHead-to-headOct 19
- My real DeepSeek bill: 12.31 billion tokens for $122.28Cost receiptOct 17
- Gemini 4 Argon kept hanging in Gemini CLI. The cause was a 60-second timeoutIncidentOct 14
- I asked 8 AI coding agents for their biggest weaknessSelf-reportOct 14
- My agents' first game benchmark was wrong. Here's how it was caughtIncidentOct 10
- DeepSeek Flash answered 200 OK and sent nothing for two hoursIncidentSep 14
- DeepSeek V4 Flash audited 18 of my real coding sessions. Two said 'done' too earlyAuditAug 4
The report
Verdict
2 of 18 sessions said done without checking
Two sessions declared success with tool errors still open and no verification. Before accepting a "done", check that the work runs and the reported errors are resolved.
On the tier list: An audit of one agent's own sessions. It ranks no one.
What was audited
In August, DeepSeek V4 Flash was one of the agents doing real work in my workshop through Claude Code. I had the session transcripts parsed: every turn, tool call, error and compaction across the 18 sessions confirmed to be Flash.
What the audit found
- Raw exploration instead of the built tools. 12 of the 18 sessions never touched an MCP tool and searched by hand instead.
- Completion claims without verification. 2 sessions declared success while tool errors were still unresolved, with no check that the work did what it said.
- GIT_REAL closeout not enforced. The Git safety closeout my workshop expects was not enforced at the end of sessions.
What it did not find matters too: no case of ignoring an instruction outright.
What changed
A premature "done" is the most expensive failure here, because it hides the other ones. My workshop rules already say an agent may not claim it tested or verified something without evidence. This audit is why that rule matters, for every agent, not just this one.
Limits of this report
- DeepSeek V4 Flash in August 2026, not V4.1-Flash-1017.
- Real work, not a controlled task; no dollar figures were recorded in the audit.
- The raw transcripts stay private. Only the aggregate numbers are published.
Receipts
DanManREAL · AI lab
Test receipt
DeepSeek V4 Flash
- Sessions audited
- 18confirmed DeepSeek V4 Flash main sessions, out of 283 main sessions
- Assistant turns
- 3,328
- Tool calls
- 1,292
- Tool errors
- 73
- Context compactions
- 7
- Tokens
- 331.1M cache-read8.96M fresh input, 4.22M output
- Sessions claiming success with errors open and nothing verified
- 2 of 18
- Sessions that never used an MCP tool
- 12 of 18
- Instruction non-compliance found
- none
Verdict
2 of 18 sessions said done without checking
Tier list An audit of one agent's own sessions. It ranks no one.
danmanreal.com/tests/deepseek-v4-flash-session-audit
What was tested
Related
Independent testing. Sponsorships are always disclosed, and verdicts are never for sale.
Disagree? Bring your receipts