5 AI claims · stress-tested
The signal-to-noise ratio in AI content is bad and getting worse. Every week brings new claims — agents will replace developers, I built a unicorn in a weekend, this model changes everything — and the ones that go viral are usually the ones that compress hardest, not the ones that are most true.
This is a running series where I take five of the loudest claims from the week and run each through one specific test in my actual engineering workflow. The methodology is short. The verdicts are uncomfortable.
claim 01 — 'agents will replace developers within 2 years'
Variation of this gets posted weekly. The strong version: by 2027-2028, software development as a profession is materially smaller because agents handle the work.
The test: I gave an agent a real ticket from a real codebase — migrate a 400-line ASP.NET MVC view to a React component, preserving validation rules and i18n. No special prompting. Just the ticket.
Result: the agent produced a plausible-looking first cut in 4 minutes. The first cut had three subtle bugs that would have shipped to production: a number-formatting locale bug, a missing edge case in the validation chain, and a state-management pattern that didn't match the rest of the codebase. All three required human-engineering judgment to catch.
claim 02 — 'I built a $10k MRR business in 30 days using only AI'
The variation pattern: someone posts a Stripe screenshot, says they used Cursor / Claude / Lovable to ship the whole thing, and the implication is that anyone could have done it.
The test: I dug into three publicly-shared "$10k MRR in 30 days" build threads. For each, I tried to find: actual revenue (not pipeline), actual users (not signups), and the time spent on non-coding work (marketing, support, audience-building) that the threads usually omit.
Result: of three threads, none cleared the bar.
- One had real $4-6k revenue, but the founder had a 40k-follower Twitter audience built over three years. The 30 days was the implementation; the distribution was 3+ years.
- One had real revenue but it was lifetime-deal money, not MRR. Different metric.
- One was actively misleading — pipeline numbers presented as revenue.
claim 03 — 'codex now beats claude code on long-running agentic tasks'
The variation: a specific tool just got an update; the update is framed as decisive in a benchmark where it previously trailed.
The test: I ran the exact same multi-step engineering task through both: a real codebase refactor across 12 files, requiring the agent to read existing code, understand the pattern, apply the refactor consistently, and update tests. Pure same-prompt comparison.
Result: Codex completed in 11 minutes with 2 reverts; Claude Code completed in 14 minutes with 1 revert. Codex was faster but produced one more bug. The difference is real but small — closer to "different trade-offs" than "decisively better."
claim 04 — 'vibe-coding produces shippable code'
The variation: a creator shows a 5-minute Cursor session generating an entire app, calls it "vibe-coded," and implies the output is production-ready.
The test: I took a real Cursor / vibe-coded app from a public demo, downloaded the code, and ran it through a basic production checklist: error handling, input validation, race conditions on the main API path, accessibility, performance under load.
Result: the demo code failed eight of the ten checklist items. Not "needed minor polish" — failed. Race conditions in the auth flow, no input validation on the public API, no error handling for the LLM call timeout case, no a11y attributes on any interactive element.
claim 05 — 'this new framework reduces tokens 50%, latency 70%'
The variation: a specific framework or technique gets posted; the numbers are dramatic; the methodology is buried.
The test: I read the published methodology behind a recent "50% fewer tokens" claim. The claim was real for a narrow benchmark: queries that fit the framework's caching pattern. Outside that pattern, the numbers were closer to flat or slightly worse.
Result: the claim is technically true but the way it's framed in the viral version makes it look general. It isn't.
what's worth your attention this week
Two things that did move the bar this week — quietly, without viral framing.
Anthropic released improvements to Claude Code's session resumption. Specifically, the ability for an agent to pick up where it left off after a context window flush. Not flashy; significantly useful for long refactors. The kind of thing you'd miss if you only watched the hot-take channels.
A specific MCP server pattern stabilized. I'd been hand-rolling agent tool connections; this week three independent implementations converged on similar patterns. The convergence is the signal — when multiple teams arrive at the same design, the design is probably right.
Neither of these will go viral. Both will matter more in 12 months than every claim above.
the methodology — and why this series exists
The series exists because the AI-content ecosystem rewards compression-friendly claims and punishes nuanced verification. A creator who posts "I built a million-dollar app in a weekend" gets 100x the engagement of a creator who posts "I shipped a useful tool in 48 hours and it's still finding its first users."
The compression-friendly claim is usually false in some load-bearing way. The verification work is unglamorous. Someone has to do it.
The standard I'm holding to:
- One specific test, in a real workflow, on a real problem.
- The numbers come from the test, not from the claim.
- Partial credit is a valid verdict — the truth is rarely binary.
- I'll be wrong sometimes; corrections welcome.
Next issue's claims, watch list:
- Agents can now autonomously close GitHub issues with PR-level quality
- The new model context window means we don't need RAG anymore
- AI-generated content is starting to beat human-written content on engagement
The verdicts on those will land where the verdicts land. The discipline is running the test, not making the claim.