OTAKAR.AI
Everything that makes the same weights run faster, cheaper, and at higher concurrency — from a single KV vector to a full serving cluster. Figures below are drawn from published model cards and inference-serving benchmarks; treat multipliers as directional, not guarantees on your own stack.
A transformer spends its inference budget in two very different ways. Prefill reads your prompt in one parallel pass — compute-bound, GPU-flops-limited. Decode generates one token at a time, each step re-reading the entire model's weights plus every cached key/value pair from every prior token — memory-bandwidth-bound. Almost every technique below attacks one side of that split.
(bytes moved from HBM) / (memory bandwidth) — not FLOPs, bandwidth| Symptom | Bound by | Sections that fix it |
|---|---|---|
| Slow time-to-first-token | Prefill compute / queueing | §6 batching, §6 disaggregation |
| Slow tokens/sec per user | Decode memory bandwidth | §2 KV design, §4 quantization, §5 speculative decoding |
| Can't raise batch size / GPU OOM | KV cache + weight capacity | §2–§3 KV design, §4 quantization |
| Model itself is too big to load | Total parameter footprint | §3 MoE sparsity, §4 quantization |
Every generated token stores a key and value vector, per layer, per head, for the rest of the sequence. That cache — not the model weights — is usually what caps your batch size and context length.
2 (K&V) × n_layers × n_heads × d_head × bytes_per_paramThree families of fix exist, in increasing order of architectural intrusiveness: share KV heads across queries (MQA, GQA), compress KV into a low-rank latent (MLA), or reduce cache precision (§4). Nearly every 2025–2026 frontier open-weight model ships one of the first two.
| Variant | KV heads | Quality vs MHA |
|---|---|---|
| MHA baseline | n_heads | reference |
| GQA (8 groups) | 8 | ~matches MHA |
| MQA | 1 | measurable drop |
| MLA (latent) | 1 latent vec. | meets or beats MHA |
| Variant | Reduction |
|---|---|
| GQA, 8 groups (128→8 heads) | ~72% |
| MLA — DeepSeek-V2 (small MoE) | ~86% |
| MLA — DeepSeek-V2 (large MoE) | ~96% |
| MLA — DeepSeek-V3 config | ~93–98% |
MLA in one line: instead of caching full-size K and V per head, DeepSeek-V2/V3 down-project them into one shared latent vector c_kv (dim 512) and reconstruct K/V on the fly at decode time via learned up-projection matrices, absorbed into the attention weights so no extra decode FLOPs are paid.
MoE decouples total capacity from compute per token: a router sends each token to a small subset of expert FFN blocks (plus, in most 2025–2026 designs, one or two "shared" experts every token visits). You pay memory for all the experts, but compute for only the active ones.
| Model | Total params | Active params | Sparsity | Routing |
|---|---|---|---|---|
| Mixtral 8x7B | 46.7B | 12.9B | 27.6% | 2 of 8 experts |
| DeepSeek-V2 | 236B | 21B | 8.9% | 6 of 160 + shared |
| DeepSeek-V3 | 671B | 37B | 5.5% | 8 of 256 + shared |
| Llama 4 Maverick | 400B | 17B | 4.3% | 1 of 128 + shared |
| Kimi K2.5 | ~1T | 32B | 3.2% | fine-grained routed |
Total params ≈ VRAM to hold the model; active params ≈ FLOPs and, roughly, decode latency per token. A 671B/37B model needs 671B-worth of memory but computes like a ~37B dense model per forward pass — the "MoE VRAM trap" for anyone sizing hardware off active params alone.
Lower precision shrinks both the weight footprint and — because decode is bandwidth-bound — the bytes that must move per step, which is usually the bigger win.
| Format | Bytes / param | Size vs FP16 | Typical quality cost |
|---|---|---|---|
| FP16 / BF16 | 2.0 | 1.0× | reference |
| FP8 | 1.0 | 0.5× | near-lossless on most models |
| INT8 | 1.0 | 0.5× | small, implementation-dependent |
| INT4 weight-only | 0.5 | 0.25× | noticeable on some tasks; strong with calibration |
| KV cache — INT8 | — | 0.5× of §2 figure | usually small |
| KV cache — INT4 | — | 0.25× of §2 figure | attention-quality risk rises |
Quantization compounds with §2 and §3: an INT4, MoE, MLA model can occupy well under a tenth of the memory of its dense FP16 MHA equivalent for a comparable active-parameter count.
(1 − α^(k+1)) / (1 − α) — α = acceptance rate, k = draft length| Method | Typical acceptance α | Reported speedup | Mechanism |
|---|---|---|---|
| Medusa | 60–80% | 2.2–3.6× | parallel heads on target model, no separate draft model |
| EAGLE-2 | 75–80% | 2.7–3.5× (70B) | autoregressive draft on target's hidden states |
| EAGLE-3 | 80–88% | 3–6.5× | draft head fused across all target layers |
| DeepSeek MTP (built-in) | >80% | ~1.8× | native multi-token-prediction head |
Below α ≈ 0.5, verification overhead can make speculative decoding slower than plain autoregressive decode — and MoE targets are harder to accelerate this way, since only the routed experts' weights need reading per step regardless of draft length.
| Technique | What it changes | Best when |
|---|---|---|
| Continuous batching | New requests join at token boundaries, not batch boundaries | almost always on — one of the highest-leverage serving wins available |
| PagedAttention | KV cache allocated in fixed blocks instead of contiguous, over-provisioned buffers | many concurrent, variable-length requests |
| Prefix / chunked prefill caching | Reuse KV state for repeated system prompts, RAG context, agent scaffolding | shared prefixes across requests |
| Prefill/decode disaggregation | Separate GPU pools for the compute-bound and bandwidth-bound phases | large, heterogeneous traffic that justifies the KV-transfer cost |
| Tensor / pipeline / expert parallelism | Split weights, layers, or experts across GPUs | model doesn't fit on one GPU, or MoE experts need spreading |
These techniques multiply, not add, and they apply to different resources — memory, bandwidth, sequential steps — so no single number represents "total speedup." As an illustration of scale, relative memory-per-token for a frontier-scale model, applying techniques cumulatively:
Bars are directional composition of the reduction ratios in §2–§4 (MLA ~93%, MoE active/total ~50% further on top of an already-sparse base, FP8 ~50%) — not a benchmark from one model. Speculative decoding (§5) and serving-layer gains (§6) stack on top of this as reductions in sequential steps and idle GPU time, not memory, so treat them as a separate multiplier on throughput.
Fit first. If the model and its KV cache don't fit in memory at your target concurrency, nothing downstream matters — reach for MoE sparsity, quantization, or MLA/GQA before anything else.
Measure the split. Separate TTFT, TPOT, throughput, and p95 latency — you cannot tell if you're compute- or bandwidth-bound without them.
Fix the actual bottleneck. Attention-kernel work (FlashAttention) matters when attention is the bottleneck, not by default — profile before optimizing.
Attack the KV cache next. It usually determines production concurrency more than any other single lever, and GQA/MLA/quantization compose.
Add speculative decoding once α is known. Measure real acceptance rate on your traffic before adopting a headline multiplier from a paper.
Optimize the cluster last. Disaggregation and exotic scheduling pay off only at a scale that justifies their operational complexity.