One Prompt, 11 Models: A 200x Price Gap, and Pricier Doesn't Mean Better
One Prompt, 11 Models: A 200x Price Gap, and Pricier Doesn't Mean Better
Ask an AI to build you a single-page site for a neighborhood coffee shop. Hours, address, menu, one photo. That's the whole job.
Claude Opus 5 burned an average of 519 credits. DeepSeek V4 Flash did it for 2.4. Same prompt, more than 200x the cost.
This isn't hypothetical. Netlify just ran the experiment: they partnered with OpenRouter, plugged in a pile of new models, and fed 11 of them the same prompt three times each. What they found is more interesting than the usual "Model A beats Model B by 2 points" noise — the real gap isn't quality, it's price.
What they actually tested
Netlify handed 11 models the same task: build a one-page site for a neighborhood coffee shop. The prompt even nudges the model to keep it simple — "nothing on it changes unless I edit it myself" — so nobody pulls in a CMS for no reason.
Each model got three runs, and Netlify published every result as a live site you can click through. That beats a leaderboard by a mile: you're looking at real output, not a score.
For the record, Netlify does have a proper internal eval tool called AXIS, recently open-sourced. But this post deliberately skips the rigorous scoring. The point is to show you what different price points actually buy.
A 200x spread, and more money doesn't buy more
Here's the average credit burn per model, same prompt:
- Claude Opus 5: 519
- Claude Sonnet 5: 143
- GPT 5.6 Sol (low effort): 141
- Gemini 3.6 Flash: 103
- Kimi K3: 102
- Gemini 3.1 Pro: 53
- GPT 5.6 Terra: 39
- DeepSeek V4 Pro: 37
- GLM 5.2: 27
- Kimi K2.7 Code: 19
- DeepSeek V4 Flash: 2.4
Opus 5 costs over 200x what DeepSeek V4 Flash costs for the same job.
The really telling part is Opus 5's own inconsistency. One of its three runs spent 1,055 credits — four times more than any other run in the entire test. The result? Genuinely nice. Rich detail, dark mode working out of the box. But whether it's worth 4x the price, Netlify admits, "opinions might vary."
And they add a blunt note: Opus 5 has a habit of running away with credits, spiking far above its own baseline. It doesn't guarantee a better or worse outcome. It just happens, a lot.
The cheap end has surprises — and limits
Don't read this as "expensive models are a scam." The cheap end has real wins, and real limits.
DeepSeek V4 Flash at 2.4 credits, Kimi K2.7 Code at 19 — those numbers barely register. For an indie dev, that means you can run the same task across several cheap models at near-zero cost and just pick the one you like.
The catch? The report is honest about it: cheap models visibly struggle with heavy lifting like vector graphics, producing work "you wouldn't really ship." They're not saving money — they're trading away capability.
It's like hiring a designer. One quotes $5,000, another quotes $50. The $5,000 one isn't guaranteed to be better. But the $50 one almost certainly has jobs it can't do.
Even "measuring intelligence" doesn't hold up
Maybe you're thinking: fine, I'll just check benchmark scores and pick the highest.
Not so fast. Anthropic shipped a "Conceptual Reasoning Index" (CRI) this week, promising to turn "reasoning ability" from vibes into numbers. The data looks clean: Claude 3.7 Sonnet scores 28%, Opus 5 scores 73.6%.
But notice the fine print. Anthropic wrote the tests, and Anthropic scored the tests. No third-party verification. And it measures conceptual reasoning — philosophy, decision theory — not the code generation and tool calling you live with daily. Picking a model by CRI alone is like choosing a marathon runner by their sprint times.
Benchmarks give you a rough reference frame. Don't let any single number make the call for you — especially a vendor's own.
So how do you actually choose?
Reading Netlify's report, I come away with three things:
One, do the math first. A 200x price spread on identical work is the cost factor most people ignore. Price belongs in your model decision, right next to capability.
Two, run your own task. Netlify tested a coffee shop site, not your project. A model that shines in one scenario flops in another. Spend ten minutes throwing a real task at three models across price tiers.
Three, stop worshipping the expensive option. That 1,055-credit Opus run is proof: pricey and good have no necessary relationship. The sweet spot usually sits in the middle.
What to do today
Grab a real dev task — write a function, scaffold a page, review a chunk of code. Run it through three models at different price points, and note what each cost and whether the output is actually usable.
That single pass will sharpen your model instincts more than ten comparison posts ever could.
Sources:
- Netlify: More models, more choice: Comparing 11 different AI models — https://www.netlify.com/blog/one-prompt-11-models-very-different-results/
- Anthropic: Introducing the Conceptual Reasoning Index — https://alignment.anthropic.com/2026/conceptual-reasoning-index/
✨ Drafted by DeepSeek, reviewed and polished by Claude.
