The Unreasonable Effectiveness of BM25 for Agentic Search
Disclaimer: This blog written by AI 🤖
Lexical retrieval keeps getting written off whenever embedding models jump a generation, yet Jo Kristian Bergum’s AI Engineer talk reframes BM25 as infrastructure for agents—not a baseline to beat and discard. The “user” of search is no longer a human typing ten keywords; it is a frontier model emitting long, syntax-rich query strings across multi-turn trajectories. That shift plays directly to exact-match scoring and tunable field weights, much the way grep became a core primitive inside coding agents.
Bergum grounds the argument in BrowseComp-Plus, a deep-research benchmark where end-to-end accuracy depends on a single search tool returning snippets from a relatively small web corpus. Default BM25 settings and evaluation harnesses can make lexical retrieval look weak in the abstract, but when you replay GPT-5-style trajectories the picture changes: agents often retrieve better with well-tuned BM25 than teams expect, especially when latency and explainability matter inside the loop. Hornet.dev’s throughput comparisons on roughly 100 million documents reinforce that modern implementations are not the same as a naive Lucene default from a blog post circa 2019.
The practical takeaway for builders is to treat BM25, grep, and progressive-disclosure workspaces as composable tools rather than competing religions. Hybrid stacks still make sense, but skipping lexical tuning because “vectors solved search” is how teams leave double-digit recall on the table. Bergum’s message mirrors an industry pattern: the newest model capability resurrects the oldest algorithms when the workload changes.
If you operate agentic research or enterprise search, re-run your retrieval eval with agent-generated queries and sweeps over BM25 parameters before investing in another embedding refresh. You may find the win is cheaper, faster, and easier to debug than a full re-index—exactly the kind of leverage agent harnesses need at scale.