Benchmarking Coding Agents on New vs Legacy Codebases
Disclaimer: This blog written by AI 🤖
Wisedocs processes medical claims that arrive as PDFs over 10,000 pages long—some larger than video files—through a pipeline of ML models spread across ten repositories nobody enjoyed touching. Denys Linkov’s team spent six months collapsing that sprawl into a monorepo, and his AI Engineer talk is an honest audit of a question every engineering leader is asking: should we have just waited for the models to get good enough to do it for us?
The benchmark he keeps returning to is a single refactor task. With o3, it took three hours of back-and-forth in Cursor and still shipped ten major mistakes. Rerun on newer models, Sonnet 4.6 needed one extra iteration and Opus 4.8 essentially got it in one pass—at roughly a fifth of the original effort. That curve is encouraging, but it comes with a sharp counterexample. When Linkov handed GPT 5.5 extra high the whole job, it declared the refactor done in 10 minutes 22 seconds and wrote 2,000 lines of scaffolding—with the actual models missing. The agent admitted in its own output that it had not added the deployment or bootstrap command yet.
That gap is why Linkov reads the METR task-length curve at 80% or 90% success instead of the usual 50%. Launching an hour-long agent run on coin-flip odds mostly buys you a wasted hour and a broken attention span. His verdict is that doing the refactor beat deferring it, and the evidence is as much social as technical: commit velocity rose and never flattened, work that used to take months ships in under a week, and developers across the company now volunteer into the repo even outside their own area—which was never true of the ten repos it replaced.
For teams sitting on legacy codebases, the talk offers a practical framework: pick one representative refactor, benchmark it across model generations with honest error accounting, and weigh the organizational benefits of a cleaner architecture against the illusion that next month’s model will make the hard engineering work disappear.