Why "Which LLM is the best?" has become the wrong question

A few days ago I sent ChatGPT off on a research job: which language model suits which task, local or in the cloud, and what to make of the new services that promise to take that choice off your hands. I dictate most of my questions, so somewhere along the way "agentic control" became "Argentinian control", and the report noted, with admirable seriousness, that it had found no literature on the subject. In hindsight that was the most honest sentence in the whole document, and it sums up the topic rather better than I would like: ask an AI a badly posed question and you get a very thorough answer to the wrong one.

The report's real point, though, was a different one. The question "Which LLM is the best?" no longer has a useful answer.

The better question

As long as there were two or three models worth taking seriously, you could pick one and live with it. Today there are several in every price range, plus open models that run on a Mac Studio, plus specialised models that have no interest in writing text at all. The question that has been gaining ground in research for about three years is therefore a different one:

Which part of my requests needs which model?

Summarising an invoice does not need a top-tier model. Finding a bug that only appears when two things in the code happen at the same time might. Send both to the same model and you either overpay for the easy one or underdeliver on the hard one, and usually both, just on different days.

Limits first, optimising second

Before quality, speed or price, there is the awkward preliminary question: are these data even allowed to go there? In Switzerland, personal data, client documents and internal figures fall under the Data Protection Act and whatever contract you have with the provider, and if you have customers in the EU, the AI Act comes along too. (In June the EU postponed its obligations for high-risk systems to December 2027 and August 2028 respectively, which is a delay and not an all-clear.)

The most common mistake is doing it the other way round: you pick the model with the nicest test scores and discover three months later that the data should never have gone there. Data protection, where the data is stored and the licence are therefore not criteria to be weighed against speed. They are limits. A model that fails them is not worse. It is simply not in the race.

Measure, don't guess

Public rankings are useful for a shortlist and fairly useless for the decision itself. They measure how models do on general tasks, and your own task is rarely general.

Two university theses show this nicely. At ETH Zurich, William West built his own test collection for his master's thesis, with datasets, prompts and automatic scoring, because no suitable public test existed for his question. At Lund University, Alexander Sae-Cho Ek and Lucas Månsson worked with Bosch to see how well a language model spots relevant articles for trend scouting. A multi-step approach did best overall, but when it came to finding the articles that really mattered, the plainer approach of simply giving the model a few examples won. And because a missed important article costs more than a superfluous one, the answer to "which is better?" depends on which mistake hurts more.

The best model, in other words, only exists relative to a mistake you have thought about in advance.

In practice that means collecting a few hundred real cases from your own work, easy and hard mixed together, and measuring three or four models from different classes against them without knowing which answer came from which. One small local model, one cheap cloud model, one strong one. It is less work than it sounds, and considerably less work than a year with the wrong model.

Cost per task actually done

The price per million tokens is the number that is easy to find, which is exactly why people get their sums wrong with it. A cheap model that botches every third answer and has to be run again ends up being the expensive one, only the bill is spread across several lines. The more honest figure is this: what does a task cost once it is actually done, including retries, checking, and the time a person spends fixing things up?

(The LLM API Price List on aiia.li shows token prices, and I stand by it. It just answers only the first half of the question.)

Passing things on: first the junior, then the senior

If different requests need different models, someone has to hand them out. Research calls this routing, and there are two basic patterns.

The first works like a well-run office. The junior looks at the job first. If he is sure, it is done; if he is not, it goes to his senior colleague. The Stanford paper FrugalGPT worked this through for language models in 2023 and measured savings of up to 98 percent at comparable quality in its experiments. That is a lab figure with the models of the day, not a promise, but the direction is clear.

In the second pattern the decision is made up front. A dispatcher looks at the request and sends it straight to the model that is likely to be enough. RouteLLM learns that decision from a large number of comparisons in which people said which of two answers was better. RouterBench provides the basis for comparing such dispatchers fairly. Both are open source on GitHub.

One warning in passing, because it belongs here: LiteLLM, a widely used open-source tool that passes requests on to many providers and sits under the bonnet of quite a few such setups, was attacked through its supply chain in March 2026. Two tampered versions were briefly available in the official Python package index. Whoever builds the point that mediates between all the models also builds the point that all the data passes through. Pinning your versions is not fussiness.

The newcomers: Jev and the Jev Router

This month two things arrived that make the whole topic a bit more tangible.

Jev from TypeSafe does not write text. It receives a situation and a question fixed in advance, with fixed possible answers, and returns one of them together with a probability. Approve or not, escalate or not, category A, B or C. Using a large chat model for decisions like that was always a bit like fetching bread rolls in a lorry.

TypeSafe says Jev cannot hallucinate. That is true if you read it carefully: a model that may only choose from given answers cannot produce invented text. It can, however, choose the wrong given answer. The form is guaranteed, the content is not, and that difference is worth testing on your own data before it quietly drops out of the sales conversation. The speed figures, too, currently come from the manufacturer alone.

The Jev Router has been available on OpenRouter since 25 September and starts precisely where this article began: for each request it picks the model and how long that model should think, weighing quality, speed and cost. That is almost word for word the answer to the better question. It is, however, two days old as I write this. New is not the same as proven, and anyone who wants to use it would do well to let it run quietly in the background first: let it decide, write everything down, compare, but not yet let it act.

What it looks like for users: Mono

What all this feels like when you just want to ask a question can be seen at Mono, a Swiss service that bundles several models under one interface. Disclosure: Mono is run by a good acquaintance of mine.

What I like about it is the openness. Right at sign-up, Mono shows which model it uses for what: GPT-6 Luna for everyday questions, Perplexity for answers with sources, Gemini Pro for large amounts of data, Claude for writing and coaching, Kimi Code for programming, Nano Banana for images. That is exactly the stage you would sensibly build first, before experimenting with dispatchers that learn: each kind of task gets its model, and anyone can see which.

You can argue about individual choices, and that is the real advantage. You can argue because you can see them. A dispatcher whose decisions stay hidden does not invite argument, it invites trust, and trust is something I have become rather more careful with where software is concerned.

Local or cloud

One short note, because the question comes up in almost every conversation with small and medium-sized businesses: open models you run yourself are not automatically cheaper. They move the bill from the provider to your own hardware, your electricity and the person who keeps the whole thing running. In return you get control, and sometimes that is exactly what the limits further up demand. For rare, difficult tasks the cloud remains the sensible default; for large volumes of similar tasks involving sensitive data, your own machine is worth a look.

One rule

If I had to boil the report, the theses and the two-day-old router down to one sentence, it would be this: don't pick the best model; decide first what must not happen, then measure on your own cases, and send every request to the cheapest model that demonstrably does the job well enough.

That is less exciting than a new ranking. It does, on the other hand, still hold next month.