Model labs

How to choose a model for production, when the best one is rarely the right one.

Lab notes on model selection. The leaderboard is the least useful input we have. Here is what we test instead, and the rule we actually decide by.

By Subhash Trivedi24 June 20265 min read

The reflex is to reach for the top of the leaderboard. Whatever model is winning the benchmarks this month, use that one. In practice we almost never do, and the reason is worth walking through, because it is the same reason so many AI projects cost more than they return.

The leaderboard stopped separating the models

As of mid-2026 there are more than a dozen frontier models a serious team can put into production. A million-token context window, which was a headline feature not long ago, is now standard across the closed frontier and matched by several open-weight models. On most common tasks, the top handful score close enough together that the ranking changes with the weather.

What has not converged is cost. Running the same task across models can differ by an order of magnitude, and at the extremes by far more. Filling the same million-token context might cost under fifteen cents on a cheap open-weight model and around ten dollars on a premium one. That is the same job, priced roughly seventy times apart. Prices move monthly, so we keep a live tracker like Price Per Token or an independent pricing comparison open rather than trusting last quarter's number.

~70×

Spread in cost for the same million-token job, cheapest open-weight to premium frontier.

40–50%

Share of the advertised context window models actually use well before quality degrades.

12+

Frontier models a serious team could reasonably ship on today.

Put those two facts together. The models are close on quality and far apart on cost. That means the interesting question is not "which is best." It is "which is the cheapest one that is good enough for this specific job." Answering it is a test, not an opinion.

What we actually test

The benchmark tells you how a model does on someone else's task. It says nothing about yours. So every dimension we evaluate is local to the job in front of us.

Fit to the specific task

Whether a smaller, cheaper or open-weight model already clears the bar for this work. Often it does, and often nobody checked. Summarizing tickets, classifying documents, extracting fields, routing requests: plenty of production work needs a competent model, not the most capable one on the market.

Cost at real volume

Per-task cost times actual volume is the number that lands on the invoice. We also watch the costs that hide: long-context surcharges, output verbosity, and retries on failed calls, which quietly double the price of anything unreliable.

Latency

For a single response it is a user-experience question. Inside an agent loop making ten or twenty calls to finish one task it compounds, and a slower model turns a two-second job into a thirty-second one.

Effective context, not advertised context

Every model benchmarked so far uses noticeably less of its window than the sticker claims. We design around what the model actually holds, not the number on the box.

Reliability and tool use

For agentic work, consistency and clean tool-calling matter more than a benchmark point. A model that is brilliant eight times and malformed twice is worse in production than one that is merely good ten times out of ten.

Data and hosting constraints

Some work cannot leave the client's environment. That alone can decide the question in favour of an open-weight model you can self-host, regardless of what tops the leaderboard.

The rule we decide by

Once those are on the table, the method is dull on purpose. Define the task and a concrete pass bar. Build a small evaluation set from real cases, not toy examples. Run the candidates against it, and always include at least one cheap or open-weight option as the challenger. Compare quality, cost, and latency together.

Pick the cheapest model that clears the bar, not the one with the highest score.

That last line is the whole discipline. Paying for capability you never use is not caution, it is a leak, and it is the same leak we described in the piece on where AI budgets disappear. A model chosen for its headroom rather than its fit produces a system that costs more per outcome for a quality difference no one can measure.

We re-run this when the models change, which is often. A challenger that lost by a hair in March may win outright by July at half the price. The eval harness is what makes that a five-minute rerun instead of a fresh argument.

Why this is a lab question and not a shopping one

The tempting version of model selection is a shopping trip: read the reviews, buy the best-rated one. The version that returns value is closer to lab work. You state a hypothesis (this cheaper model is good enough for this job), you test it against real cases, and you let the result decide. That connects directly to the argument in why AI ROI is an organisational problem: the return does not come from owning the most capable model. It comes from matching a good-enough model to a well-defined job and measuring whether it clears the bar.

The model itself is the cheap, swappable part of the system. It will be beaten and repriced within months. The durable assets are the evaluation set that defines "good enough" for your work and the judgement to stop paying for more than the job needs. Those do not expire when the leaderboard reshuffles. Building them is most of what we do, and it is described in how we work.

The best model is rarely the right one. The only way to know which is, is to test.

We will test your job against a cheap challenger.

Every build we ship comes with the evaluation set behind it, so the model stays the swappable part.

What we build