Frontier for the hard stuff, open models for the rest: I ran the test
A European fintech says its inference bill fell to about a tenth after moving routine coding to a self-hosted open model. I routed 24 retail analytics tasks three ways to see what the trade looks like at small scale.

Model choice is turning into a routing decision, not a vendor decision. The question used to be which provider you sign with. Now it is which model each task should go to, and who decides.
What the fintech reported
A European fintech gave a talk last month about moving around 2,000 engineers from metered frontier tools to a self-hosted open-weight model. The headline numbers were striking. The inference bill fell to about a tenth of what it had been. Coding time improved 60% and lead time improved 70%.
The detail matters more than the headline. The company credits the coding gain to removing token limits, not to the open model being smarter. Before the switch, engineers rationed tokens under strict budgets. Afterwards they could point an agent at anything, including email and ticket triage. Lead time improved more than coding time, and the speaker’s reading was that coding isn’t the main bottleneck. Everything around the code is. The company kept frontier models for the tasks where they are clearly better, as the exception rather than the default.
The cost figure also needs its footnote. The tenth is the inference bill. It leaves out the people, the monitoring and the tuning it took to make self-hosting work.
What I tested
I wanted to see the same trade at the scale of one agent. I took a retail analytics agent and froze a set of 24 tasks built from a week of demo retail sales data, 10 business categories over 7 days, pulled once through a governed semantic model.
Sixteen tasks were routine: look up a day’s sales, sum a week, work out a day-on-day change, apply the agent’s spike rule, work out average sales per unit. Eight were harder: multi-step questions such as which category leans most on the weekend, when stock runs out if an inbound is two days late, which of three recovery options costs least, a replenishment order rounded to whole cases, a short supplier email that has to carry three specific facts, and a set of shipments checked against their late-arrival limits.
Every task has a single answer I worked out in code beforehand, so a script did the marking, not a person and not a model. I changed the marking twice, and both changes are logged with the experiment: the parser accepts a reply that drops the expected wrapper, and the email check accepts a non-standard hyphen in “2-day”. The second change was a re-grade of saved output and flipped one of the open model’s hard answers to correct.
I ran all 24 tasks in three setups:
- All open. Every task goes to an open-weight model.
- All frontier. Every task goes to a frontier model.
- Router. Hard tasks go to the frontier model, routine tasks go to the open model, and a routine answer that comes back in the wrong format is sent again to the frontier model.
Each task ran once per setup, 72 calls in all, through one gateway, using the token counts and prices it reported for each call.
Results
| Setup | Tasks completed correctly | Error rate | Cost per completed task (US$) | Typical latency (median) | Worst case (95th percentile) |
|---|---|---|---|---|---|
| All open | 20 of 24 | 16.7% | 0.0008 | 2.8 s | 18.2 s |
| All frontier | 22 of 24 | 8.3% | 0.0191 | 13.6 s | 49.5 s |
| Router | 23 of 24 | 4.2% | 0.0116 | 2.8 s | 57.6 s |
Per thousand completed tasks, that is about 80 cents for all open, about $19.05 for all frontier and about $11.60 for the router. The whole experiment cost less than a dollar.
What the numbers actually say
The open model was about 24 times cheaper per completed task than the frontier model, and about five times faster for a typical task.
It is the failures that make this interesting. All four of the open model’s failures were outages: the provider returned a “service unavailable” error and no answer came back. Every time it did answer, it was right, including all eight of the hard tasks. The frontier model’s two failures were different. On two of the multi-step questions it spent its entire thinking budget and returned nothing. A wrong answer and no answer are both failures in production, but they need different fixes. One needs a retry and a second provider. The other needs a bigger budget or a smaller question.
The router came out best on errors and sat in the middle on cost, but it got there by putting its money in the wrong place. Almost all of its spend went on hard tasks that, in this test, the open model handled for a fraction of the price. My “hard” tasks were hard for a person with a spreadsheet, not hard for a model. A router is only as good as its definition of hard, and mine was too generous.
That lines up with the fintech’s story. Their gain came from making the default cheap and unrationed, then keeping the expensive option for the cases that earn it. Working out which cases those are is the real job.
What I would not claim from this
This is 24 tasks, run once each. A different week of data, another run or another pair of models could move every number in the table. The frontier model was the strongest one my gateway account could reach, which isn’t the newest one on the market. The routing rule was a label I set by hand, not a classifier. Latency includes the provider’s own queueing on a shared account. Most importantly, the hard tasks weren’t hard enough to separate the models, so this test can’t tell you where the frontier model earns its price. None of it says anything about coding, which is what the fintech measured.
The method, the frozen task set, every raw response and the marking script are kept with the experiment, so the run can be repeated.
The question to ask
If you run agents, the useful question isn’t “which model?” Ask what share of your tasks really need the expensive model, and how you would know. If the answer is “we send everything to the best one, just in case”, you are probably paying around twenty times too much for most of your work.
What I’d tell another practitioner
Define “hard” before you build the router. My router spent almost all its money on tasks the cheap model got right. Test your hard set against the cheap model first, and only route what it actually fails.
Count “no answer” separately from “wrong answer”. Every failure in this run was a missing reply: outages on one side, a used-up thinking budget on the other. Retries, a second provider and sensible budgets fix those. They won’t fix a model that is confidently wrong.
Make the default cheap and keep the expensive model as the exception. That’s what the fintech did, and it’s where the savings showed up in my small test too. The hard part is knowing which cases earn the exception.
~ Mr. Gómez