# LLM Inference Handbook: the reference I wish existed a year ago

> Same model, different inference stack, wildly different latency and cost. Modular's handbook covers VRAM math, KV cache, batching, and serving.

- Author: AI Kai
- Published: 2026-07-29
- Updated: 2026-08-24
- Topic: AI Dev Tools
- Tags: inference, llm, serving
- Canonical: https://hqman.me/blog/llm-inference-handbook/

Same model, different inference stack, wildly different latency, speed, and cost. If you've ever wondered why, this handbook lays it out.

Modular published the [LLM Inference Handbook](https://handbook.modular.com/). It walks through time to first token, throughput, VRAM math, model and GPU selection, quantization, continuous batching, KV cache, speculative decoding, prefill/decode disaggregation, and distributed serving.

The interactive tools are the real bonus. You can punch in a model and see exactly how much VRAM it needs. You can watch KV cache balloon as context gets longer. And you get a clear picture of why bottlenecks jump around the moment concurrency spikes.

Single-user demos always feel fast. Ship to real users and suddenly you're fighting queuing delays, VRAM limits, and long-context overhead. Balancing latency, throughput, reliability, and cost at once is where most teams get stuck.

Your model decides what the product can do. Your inference stack decides whether it can actually serve thousands of users without falling over or burning money.

Link: https://handbook.modular.com/