The State of Model Routing: Why Benchmarks Lie in Agent Sessions
Disclaimer: This blog written by AI 🤖
The obvious way to save money on LLM inference is to route each task to whichever model scores best on a benchmark. A panel from NVIDIA, Cognition, and OpenRouter argues that this breaks down the moment you deploy agents. Alex Atallah of OpenRouter points to terminal bench: Haiku is far cheaper per token, yet Opus completes the suite at roughly three times the quality for a tenth of the total cost. Push a small model outside its training distribution and it thrashes—calling tools in loops until it burns more budget than the expensive model ever would.
Walden Yan of Cognition calls naive task-based routing fragile for a deeper reason. An agent session rarely stays one kind of task. It starts as a question about a codebase, becomes a feature request, then devolves into live debugging. The model you picked at minute one is stranded by minute twenty. Cognition’s Devin Fusion keeps a frontier model planning and delegates implementation to cheaper workers, cutting the cost of Fable-level intelligence by 40% while going deeper—because a cheaper model can afford to spin off three sub-agents to explore a repo in parallel.
The systems details matter as much as the model choice. Cognition avoids spawning fresh sub-agents in favor of one sidekick with continuous context so the KV cache stays warm; cached tokens cost roughly ten times less than fresh ones. Yan argues compaction should be driven by intelligence needs, not cost savings, because compacting forces a cache miss and quality falls off a cliff well before the advertised million-token window. OpenRouter’s auto router sat almost unused for two years until OpenClaw began sending heartbeats every ten minutes—one popular app with two completely different intelligence profiles finally made routing a product instead of plumbing.