Deep Dive on LLM Inference at Scale: KV Cache Math, Flash Attention, and Engine Selection
Disclaimer: This blog written by AI 🤖
Running LLM inference at scale is less about picking the biggest model and more about understanding where memory, latency, and throughput collide. In this AI Engineer workshop, Harshul Jain — a senior software engineer at Audible — and Tanmay Sah, an independent AI researcher, build the cost model from first principles. They start with a concrete number: a single token of KV cache on Mistral 7B costs 131 KB. Multiply that by 16,000 tokens of context and 80 concurrent users, and the cache alone demands 42 GB of GPU memory — which is why requests start failing on a 24 GB card.
The session maps three symptoms every production team eventually hits: memory that climbs with context length, time-to-first-token that degrades as prompts grow, and throughput that collapses when a naive server processes requests one at a time. From there, the workshop walks down through the inference pipeline into the attention layer and explains how a GPU splits memory between fixed model weights and the cache that must grow with every token. Prefill and decode phases get separate treatment, with decode identified as the memory-bound bottleneck that dominates real-world serving.
Optimization splits cleanly into two tracks. Sah covers the model side — quantization, multi-head through grouped-query and latent attention variants, flash attention, and tiling — using deliberately playful teaching frames (the “ostrich algorithm” for assumptions you wave through, the “world cup algorithm” for bracketing hard problems). Jain covers the serving side: paged attention borrowed from OS paging, continuous batching, prefix caching, and KV quantization, each benchmarked against a plain baseline. They close on engine selection, where standard workloads showed no statistical difference between vLLM and SGLang, but agentic branching workloads opened a three-to-four-times gap. Slides and runnable notebooks are linked in the workshop repo for teams that want to reproduce the benchmarks.
For harness engineers and platform teams, the takeaway is that inference cost is a systems problem with a calculable floor. Before adding another GPU or switching models, run the capacity math on your actual context lengths and concurrency targets — then pick optimizations on the side of the stack where your bottleneck actually lives.