Finding signal on Twitter is more difficult than it used to be. We curate the best tweets on topics like AI, startups, and product development every weekday so you can focus on what matters.
Originally published by @clairevo on X. Tech Twitter preserves the original source alongside this readable edition.
Before you dive in
• Discover why GPT-4.5 won ChatPRD's 82-build AI model bake-off
• Cheaper coders failed; frontier models reign
• Planner architecture matters more than expected.
Best for builders who want practical takeaways. 6 min read.
I'm working on integrating a full prototyping tool into ChatPRD. This prototyping tool follows a 3 step agent pattern:
a planner designs the architecture
a coder writes the files
and a visual reviewer grades the result.
Every one of those three steps is a model we get to choose. So I ran a real bake-off — 82 sandbox builds across planners, coders, themes, and product types — to answer one question: is “smarter planning plus cheaper coding” actually a good trade?
TL;DR. Despite my initial love of GLM-5.2, cheaper coding is a trap. Every budget coder I tried either fell over or burned 2–6× the tokens.
GPT‑5.5 is the clear winning coder: 10/10 functional, zero hard failures, and the leanest token bill (~$0.50 a build). And as expected: the planner, not the coder, turned out to be the most interesting lever, but only when the product demanded complexity.
How we ran our evals
For each build, we ran the full pipeline in an isolated cloud sandbox: external planning → coding → static checks (typecheck/lint/build) → browser validation (it actually clicks the primary interaction and screenshots desktop + mobile) → visual review.
I defined a test set of prototypes for the experiment:
support inbox
habit tracker
content publishing
inventory ops
incident response
And execute these in two themes: a grayscale wireframe and a polished shadcn app.
Of course, to grade 82 builds, I didn't eyeball them by hand.
Claude Code ran a fan-out of inspector agents. One per build, each run reviewed the generated code and looked at every screenshot. Then, a second adversarial verifier tried to refute each verdict against the source code.
This worked pretty well: the verifier caught the visual reviewer flagging “missing” features that were actually there, just hidden behind a default state the screenshot never reached.
We doomscroll, you upskill
Get the 10 tweets shaping how builders think today.
Newsletter
We doomscroll, you upskill
Get the 10 tweets shaping how builders think today.
Before the scores, I want to show you the output! These are real, unedited builds. On the left, a polished inventory console; on the right, a wireframe editorial board — same engine, different theme and product.
📷Inventory ops · polished theme PASS · visual 82
📷Content publishing · wireframe PASS
The coders: GPT‑5.5 wins, cheap loses
We pitted five coders against our current default (gpt-5.4), all driven by the same GPT‑5.5 planner. The pattern was stark.
CoderFunctionalHard failsAvg tokensVerdictGPT‑5.510/100128kWinnergpt‑5.4 (current default)9/100139kTied on qualityClaude Sonnet 4.65/101234kSlow (346s/build)Qwen 3.7 Plus5/103486kToken-hungryClaude Opus 4.84/105483kWorst & priciestGemini Flash / Flash‑Lite / GLM~0mostup to 685kDid not converge
The “value” pitch — a cheaper model writes the code — never materialized. Budget coders didn't just score lower; they failed to finish and burned more tokens doing it. Opus was the cautionary tale: half its builds never produced a working app, and at its tier rate it accounted for roughly $88 of the study's ~$121 model bill all by itself. The takeaway: the coder has to be a frontier model, and among those, GPT‑5.5 was the most reliable and the leanest.
The recurring weak spot — mobile. Almost every coder nailed the desktop layout and then fumbled the phone. The classic failure: a support inbox that's a proper two-pane workspace on desktop but, on mobile, drops you straight into one conversation with no way back to the queue.
📷Desktop: a credible queue + thread + activity workspace.
📷Mobile: straight into a conversation — no queue, composer below the fold. quality miss
The plot twist: the planner matters more than we thought
Our first 40-build pass concluded “smarter planning doesn't move quality.” That was too hasty — we'd only compared two planners. When we added Claude Opus 4.8 as a planner (holding the coder at GPT‑5.5), something jumped out on incident response.
Asked to build the same incident tool, GPT‑5.5's planner produced a single console that leads with a metrics row. Opus's planner instead designed a two-route master–detail app: a queue, and a dedicated incident page that drops you onto the next action. On mobile, that architecture difference is the whole ballgame.
📷Opus‑planned: straight to the incident, “Resolve” above the fold. PASS · visual 86
📷GPT‑5.5‑planned: competent console, but KPIs push the work down. visual 71
It held up under repeated runs (86/86/84/84) — a real, reproducible win driven by better architecture, not luck. But before crowning Opus the universal planner, we stress-tested it…
…and it didn't generalize. On the wireframe theme, the Opus advantage vanished — GPT‑5.5's planner was equal or better on every case. And the habit-tracker “win” turned out to be single-sample noise (it passed only ~half the time on a re-run). The honest conclusion: a smarter planner helps specific product shapes — list→detail flows especially — but it is not an across-the-board quality dial. Confirmation passes exist precisely to deflate good-looking results, and this one did its job.
For balance, here's the habit tracker the Opus plan produced — genuinely clean, even if the “lift” over GPT‑5.5 wasn't statistically real:
📷Daily habits · progress ring, streaks, list above the fold.
One more contender: the grader itself
The visual reviewer is a model too, so we replayed eight builds through three of them. The cheaper Gemini reviewers consistently inflated scores by 10–20 points and waved through genuinely broken UIs (one gave a failing build 92/100). Our incumbent gpt-5.4 reviewer was the most discriminating — it catches the borderline cases that actually matter for a quality gate — so it stays.
What we're shipping
The bake-off makes the coder call easy: GPT‑5.5 is the winning coder — most reliable, leanest, ~$0.50 per build. Everything else we learned is about not over-reaching:
Coder → GPT‑5.5. Frontier-only; cheap coders are a false economy.
Visual reviewer → keep gpt-5.4. The cheap graders are too generous to gate on.
Planner → smarter helps, selectively. Opus-style planning is worth it for list/detail-heavy products; not a blanket upgrade.
Guardrails that paid off: a route-contract validator (no links to routes that don't exist) and a screen-scope limit kept the whole fleet honest across 82 builds.
The number that decided it. An average GPT‑5.5 build costs ~$0.50 (~$0.46 coding + $0.04 planning + $0.02 review) and passes every functional check. The cheaper alternatives cost more once you count the retries — reliability is the real budget line.
Method notes & honesty box: 82 sandbox builds; quality scores come from a model reviewer with a known blind spot for state-gated UI, so single-point quality gaps were treated as ties; Gateway per-token pricing isn't fully wired, so non-OpenAI dollar figures use current tier-rate analogues (the relative story is solid, the absolute dollars are estimates). Full evidence ledger lives in docs/features/functional-prototype-v2-model-routing-evaluation.md.
