How I test AI tools

I test AI tools on my own projects and show what happened: what worked, what failed, how long it took and what it cost. Here is how to read the results.

Tests run on real work

Every model, coding agent and tool I test does real work from my own projects: the same repositories, deadlines and mess as any working developer. Tests take real time, and local tests take real GPU hours. My local tests run on an RTX 5080 (16 GB) and an RTX 5060 (8 GB).

Every claim has a receipt

A test publishes what it can show: logs, actual costs, timings and failures. Every report ends with a verdict and a plain list of what it did not measure. When a source is private, I publish the numbers, not the private files, and I never publish credentials or unrelated private information.

What kind of evidence it is

Every report carries a label for the kind of evidence behind it:

• Head-to-head Two or more tools or workflows on the same real task, under the same conditions.
• Field test One tool doing real work, with what it did logged.
• Incident Something an agent or a provider actually did, verified afterwards.
• Audit A measured review of real sessions or runs.
• Self-report What an agent says about itself. Published word for word, labelled, never ranked.
• Cost receipt A real bill or usage record.

What never counts as a test

• Vendor benchmark tables. I show them as labelled context, never as my result.
• Self-reports. What an agent says about itself is interesting, but it is not performance.
• Pricing pages and other people's coverage. News stays on YouTube; this site gets tested posts and receipts.

How the tier lists work

My personal placements reflect daily use on real work. Published reports show the evidence I have, including its limits; a personal tier is not a controlled benchmark result. The AI Coding Agents Tier List is the flagship. Its tiers mean:

• S: Hand it real work and expect a verified result.
• A: Strong on real work; plan a review pass.
• B: Useful with close supervision.
• C: Only for narrow, well-fenced tasks.
• D: Not recommended for real work right now.

Controlled comparisons use the same task, starting repository state and time box, with logs, costs, failures and review results. Major releases are reasons to revisit the evidence; they do not establish a new result by themselves.

When something better turns up mid-test

If a credible upgrade appears while a test is running, the test follows it, and the evidence for both the original work and the upgraded test stays published.

Community receipts

On Discord you can request a test, argue with a placement or share your own receipts. Community results are welcome, and they stay labelled as community submissions, never presented as tests I ran.

Independence

Sponsorships fit where I show AI tools in real use, and they are always disclosed. Verdicts are never for sale. I also offersetup services with a defined scope; buying one has no effect on a verdict.

If I got something wrong

Bring the evidence to Discord. If a receipt is wrong, the page gets corrected, and the change is noted on it.