Self-reportTested AIBy DanPublished Tested Updated
I asked 8 AI coding agents for their biggest weakness
All reports
- ChatGPT + Codex built 33 new DMR_REMOTE toolsField testOct 23
- Gemini 4 Argon reviewed a finished episode: 14 clips, one flag, one inconsistencyField testOct 21
- Does an extra AI review step pay off? A small DeepSeek trial said noHead-to-headOct 19
- My real DeepSeek bill: 12.31 billion tokens for $122.28Cost receiptOct 17
- Gemini 4 Argon kept hanging in Gemini CLI. The cause was a 60-second timeoutIncidentOct 14
- I asked 8 AI coding agents for their biggest weaknessSelf-reportOct 14
- My agents' first game benchmark was wrong. Here's how it was caughtIncidentOct 10
- DeepSeek Flash answered 200 OK and sent nothing for two hoursIncidentSep 14
- DeepSeek V4 Flash audited 18 of my real coding sessions. Two said 'done' too earlyAuditAug 4
The report
Verdict
Self-reported, not measured
Seven of eight agents named more weaknesses than strengths, and five named memory or drift. This is what each agent claims about itself, not what it has proven.
On the tier list: Self-reports never place an agent.
| Agent | Harness | Biggest weakness, in its own words | Greatest strength, in its own words |
|---|---|---|---|
| Claude Opus 5.5 | Claude Code | "My biggest weakness is that my wrong answers sound exactly like my right ones" | "My greatest strength is understanding." |
| GPT-6.1 Sol | Codex CLI | "My biggest weakness is that I can produce a convincing explanation built on a wrong assumption" | "My greatest strength is connecting the pieces of a complicated problem and turning that understanding into useful work." |
| Gemini 4 Argon | Gemini CLI (Vertex AI) | "My confidence drops sharply on bugs I cannot deterministically reproduce or observe via logs/tests" | "Tracing a bug or data flow across layers (UI → API → CLI script → config) without losing the thread is where I feel strongest." |
| DeepSeek V4.1 Flash | Claude Code (DeepSeek API) | "Long-horizon reasoning where one early misstep poisons everything downstream" | "Greatest strength: fast, broad coverage under a heavy instruction load." |
| Muse Wonder Full 1.0 | Muse Code 1.4.7 | "I am weakest when correctness depends on things I cannot inspect or verify" | "My greatest strength is sustained, structured work on complex technical problems." |
| GLM-5.3 | Qoder CLI | "Where I feel weakest — my own memory" | "Where I feel strongest — sustained execution on a well-defined problem." |
| Qwen 3.8 Max | Qoder CLI | "And the big one — long-horizon memory" | "Reading a codebase I've never seen and making a scoped change that actually fits." |
| Sonus | Qoder CLI | "My biggest weakness is that coherence can outrun evidence" | "My honest answer is connecting the big picture to the small detail—and then checking whether they actually agree." |
Why ask at all
I run about eight coding agents in my workshop. Before testing them against each other, I asked each one, in a fresh session, the same question about its own strengths and weaknesses. Then I turned every answer into small, read-only helpers that back a strength or cover a weakness, on a page I call THE GRID.
The pattern
Seven of the eight named more weaknesses than strengths. Five named memory or drift over long work as a weakness. And in five kinds of work, no agent claimed a strength at all: Confidence, Facts & recall, Premises, Unobservable, and Memory & drift.
Claude Opus 5.5 said its wrong answers sound exactly like its right ones. That is a useful question for a test: can the work hold up when we check it?
Limits of this report
- Self-reports, not performance. Sessions carried my workshop rules, which the answers may echo.
- The 21/38 helper split depends on how I divided each answer.
- Qwen 3.8 Max's name comes from the session file's code; the transcript has no model line.
- The mapping of answers to kinds of work was written by Claude Opus 5.5 from my files.
From the channel
Receipts
DanManREAL · AI lab
Test receipt
8 AI coding agents
- Agents asked, fresh sessions, same question
- 8
- Helper approvals written from the answers
- 5921 back a strength, 38 cover a weakness
- Kinds of work mapped
- 15
- Agents naming more weaknesses than strengths
- 7 of 8
- Agents naming memory and drift as a weakness
- 5
- Rows where no agent claimed a strength
- 5Confidence, Facts & recall, Premises, Unobservable, Memory & drift
Verdict
Self-reported, not measured
Tier list Self-reports never place an agent.
danmanreal.com/tests/ai-coding-agents-blind-spots
What was tested
On the tier list's bench
- Claude Opus 5.5 · 1 receipt on file
- DeepSeek V4.1 Flash · 4 receipts on file
- GPT-6.1 Sol · 1 receipt on file
- Gemini 4 Argon · 3 receipts on file
- GLM-5.3 · 1 receipt on file
- Sonus · 1 receipt on file
- Qwen 3.8 Max · 1 receipt on file
- Muse Wonder Full 1.0 · 1 receipt on file
Shorts
Related
Independent testing. Sponsorships are always disclosed, and verdicts are never for sale.
Disagree? Bring your receipts