OpenRouter’s New Routers: The Same API Name Is Not the Same Test

On November 6, OpenRouter announced a rebuilt Auto Router and a new coding-focused Auto Code router. Existing users can keep the same openrouter/auto name. That continuity is convenient, but it creates an important buying question: what exactly did your pilot approve?

Athelguard’s view: an unchanged API name is not evidence of unchanged behavior. When a service chooses among models, the thing being assessed is a selection policy operating over a changing pool, not just one model.

A successful request can conceal a changed assumption

OpenRouter’s current documentation says the previous pin_model and cost_quality_tradeoff settings can still be accepted even though the current router does not use them. It also says the default policy costs more per task than the previous generation; low is the suggested setting for costs closer to previous behavior. These are documented design choices, not findings of a defect.

For a buyer, the practical lesson is narrower than “automatic routing is risky.” An integration test that checks only for a successful response can miss whether a purchasing assumption still holds. A setting present in a configuration file is not proof that it is controlling the service.

The distinction in the diagram matters: a permitted route can still produce an unacceptable result, and a good result does not prove that the intended restrictions were enforced.

Test sessions, not just isolated answers

OpenRouter documents that the selected model may change between turns and that switching can lose prompt-cache benefits. A one-prompt comparison therefore leaves out part of the operating cost. The response’s model field identifies the model that answered; preserve that evidence alongside the settings used for the test.

Consider a hypothetical software team buying a routed coding assistant. A useful acceptance exercise would start from the same repository state, give each candidate the same bounded task and require the same tests to pass. Count retries, failed attempts and human repair time, not only the first answer’s token bill. Run more than one session before treating a result as representative.

Keep three records:

• The approved setup: dated model and provider restrictions, cost preference, tool permissions and task acceptance criteria.
• The actual run: the route taken on each turn, usage, errors and any missing evidence. Do not fill gaps with assumptions.
• The decision: whether the completed task met the agreed quality, time and spending limits, and which changes should trigger another check.

This does not require freezing every model forever. It means deciding which changes can be accepted automatically and which require a small repeatable test. Avoid capturing confidential prompts merely to make the record more complete; collect only the evidence needed.

How Athelguard can help

An Independent Technical Assessment can help turn “this AI service worked in a demo” into explicit acceptance questions: which settings matter, what evidence would demonstrate them, and what would justify retesting. The objective is a defensible buying decision, not a promise that one router or model will remain best.

This article analyzes published documentation. Athelguard has not independently tested these routers or verified the vendor’s performance claims.