Serving one user at a time, Gemma-4-26B decodes at 127 tokens/s on an A100 straight out of the box. Two levers took it to 297, 2.34× faster, with the exact same output: quantizing the weights to FP8, and switching on a speculative-decoding head the model already ships but vLLM had disabled. Neither was the first thing tried. A custom kernel in between did nothing at all, and the larger of the two levers first measured as a loss. This is the whole path, dead-ends included.
The same answer, decoded live in two panes at each configuration’s measured per-token latency. Hit replay and watch the gap open up.
What batch-1 latency actually is
Batch-1 decode latency is the number a person feels: the speed a chat reply streams back, one token at a time. Throughput benchmarks at high concurrency measure a different quantity, cost per token across many simultaneous users. For a single interactive stream the metric is tok/s for one request, and that is set entirely by how fast the next token appears after the last.
The workload is google/gemma-4-26B-A4B-it, a fine-grained mixture of experts: 25.2B parameters total,
3.8B active per token, 128 experts with the top 8 routed plus one shared, on a single A100-80GB running
vLLM 0.25.0. Fine-grained is the operative word. This is the DeepSeek and Qwen3 family, not Mixtral,
and that structural difference decides which optimizations pay and which are traps.
The exact architecture, and why it rules out 4-bit
Gemma-4-26B-A4B is 25.2B total parameters, 3.8B active per token. Each layer routes to the top 8 of 128
experts plus one always-on shared expert; the expert FFN width is N=704, the hidden width 2816, over
30 layers. Attention interleaves 25 sliding-window layers to 5 global (a 5:1 ratio), the vocabulary is
262,144 with a tied lm_head, and the context window is 256K. It ships a 430M Multi-Token-Prediction
drafter alongside the main weights.
The 704-wide experts are the reason Google recommends INT8 or FP8 rather than 4-bit for this model: 4-bit group quantization needs wider matrices to stay accurate, and 704 is too narrow. That single fact removes the most aggressive quantization option before any measurement.
First principles: compute-bound or memory-bound?
Before any code ran, one question decided the entire search: is a decode step limited by arithmetic or by memory? A batch-1 decode step multiplies each weight matrix by a single vector. That is a GEMV, and a GEMV does roughly 2 FLOPs for every byte it reads, an arithmetic intensity near 1 FLOP/byte. The A100’s BF16 ridge point, where compute and bandwidth are balanced, sits around 153 FLOP/byte. Decode runs about 150× to the left of that ridge. It is memory-bandwidth-bound with no ambiguity, and the consequence is strict: latency is set by how many weight bytes stream per token, not by how fast the cores can multiply. That one result prunes a whole class of ideas before any code runs: a faster matmul kernel, tensor-core tricks, a higher-FLOP GPU all miss a bottleneck made of bytes moved, not math done. Every lever that survives has to move fewer bytes per token.
Why an arithmetic intensity near 1 forces a bandwidth-bound step
Arithmetic intensity is useful work per byte moved. A matrix-times-vector reads every element of the matrix once and does one multiply-add per element, so it does about 2 FLOPs per 2-byte weight, roughly 1 FLOP/byte. The roofline model says a kernel is bandwidth-bound whenever its intensity is below the hardware’s ridge point (peak FLOP/s divided by peak bytes/s). For the A100 in BF16 that ridge is near 153 FLOP/byte. At 1 FLOP/byte the step spends essentially all its time waiting on HBM, and the processing units sit idle between reads. Batching many requests together raises the intensity (each weight read is reused across requests) and eventually crosses the ridge, but a single interactive stream never does.
So the question became: how many bytes, and where do they go? Read straight from config.json weight
shapes, one token in BF16 streams 7.36 GB. On the A100’s 2.04 TB/s of HBM that is a hard floor near
3.6 ms/token, a ceiling around 278 tok/s that no kernel can beat. Those bytes split like this:
- experts38.8% · 2.86 GB
- attention26.3% · 1.93 GB
- lm_head20.1% · 1.48 GB
- dense MLP14.5% · 1.07 GB
- router0.3% · 0.02 GB
Two facts fall straight out of that table, and both prune the search. Routed experts are only 38.8% of
the traffic, not the 80% a Mixtral-shaped intuition assumes, so optimizing only the experts caps the
win early. And the tied 262K-vocab lm_head is a hidden 20.1% tax that most quantization recipes leave
in BF16. Because the step is bandwidth-bound, weight-byte share is a direct proxy for latency share. And
the bytes read per token are just bytes-per-pass times passes, so the roofline’s one instruction, move
fewer bytes, factors into exactly two lever families and no third: make each pass read fewer bytes
(quantize the weights), or make fewer passes for the same output (speculative decoding). Nothing else
touches the bottleneck.
Where the 7.36 GB goes, weight matrix by weight matrix
The count is a hand calculation from config shapes, not profiler output, so it is a roofline proxy rather than a trace. Per token, in BF16 (2 bytes each):
- Experts, 2.86 GB (38.8%). Top-8 of 128 experts, three matrices each (gate, up, down) at 2816×704, across 30 layers: about 1.43B parameters read.
- Attention, 1.93 GB (26.3%). The q/k/o projections under grouped-query attention, 25 sliding layers plus 5 global: about 966M parameters.
lm_head, 1.48 GB (20.1%). The tied output projection at 2816×262,144: about 738M parameters, read once per token to produce logits over the full vocabulary.- Dense MLP, 1.07 GB (14.5%). The always-on shared MLP at 2816×2112 over 30 layers: about 535M.
- Router, 0.02 GB (0.3%). Negligible.
The proxy deliberately omits KV-cache reads and the per-launch overhead of thousands of tiny kernels, which is why the achieved 7.66 ms/token sits above the 3.6 ms theoretical floor rather than on it.
One more thing about the baseline, because it decides what counts as a real win. vLLM’s default already runs under CUDA graphs, and on this model that alone is worth 9.4×: eager execution decodes at 13.5 tok/s, the graph-captured default at 127.4. A fine-grained MoE step launches thousands of tiny kernels per token (a router, eight expert GEMMs, attention gather and scatter, across 30 layers), and in eager mode the per-launch Python overhead swamps the actual math. Graphs replay the whole step as one recorded unit. The default has already banked the single biggest speedup available, so every number below is measured against that 127.4, not against eager. Live measurement then confirmed the analysis: per-token latency on the validation run was dead flat from p50 to p99 (7.73 to 7.76 ms), the exact signature of a bandwidth-bound step where every token does identical work. At 7.66 ms the greedy default is running at about 47% of the 3.6 ms floor, and that gap is the headroom the ladder claws back.
Lever 1: fewer bytes (FP8 weights)
Of the two families the factorization named, fewer bytes per pass is the one to try first, because it acts on all 7.36 GB at once instead of a subset. FP8 weight-only quantization pulls it directly: store every weight in 8 bits instead of 16, so each pass reads half the weight bytes.
Getting there was not clean. The obvious route, downloading a ready-made RedHatAI FP8 checkpoint,
dead-ended twice: first anonymous Xet throttling, then an Auth SignatureError: invalid key pair id.
The workaround was simpler than the download. Passing --quantization fp8 quantizes the BF16 model at
load time and routes it to a weight-only FP8-Marlin kernel, no checkpoint fetch at all.
That bought +17%, from 127.4 to 149.5 tok/s greedy, near-lossless. A halve-the-bytes lever returning only 17% sounds underwhelming until the byte table explains it exactly.
Why halving the weight bytes bought only 17 percent
The A100 has no FP8 tensor cores; that hardware path starts on Hopper and Ada (sm_89 and up). So
FP8-Marlin dequantizes each weight back to BF16 in registers and does the matmul in BF16. The compute
is unchanged, and the only saving is the trip out of HBM, half the bytes for the tensors it covers. It
also does not cover everything cleanly: on Ampere the experts, 38.8% of the traffic, are the part FP8
accelerates least, so the realized gain lands right where the byte budget predicted. The +17% is not a
disappointment, it is the budget being obeyed.
(An earlier run against a slightly different 119.7 tok/s baseline read this lever as +12%, at 133.8 tok/s. Same kernel, different denominator. The +17% is measured against the coherent-prompt 127.4 default used everywhere else on this page.)
One adjacent lever was ruled out by the hardware rather than by choice. Quantizing the KV cache to FP8
would cut the KV bytes too, but on the A100 it killed the server outright: FP8 KV cache needs sm_89
and up, the same missing capability. Dropped, not claimed.
Quantization alone was never going to reach 2×, and the byte table said so up front: experts plus the tensors FP8 helps least cap it near +17%. The other lever family, fewer passes, had to carry the rest.
The kernel that did nothing
Before speculative decoding, one detour looked like the differentiated move: write a better MoE kernel. vLLM ships hand-tuned fused-MoE configs for specific expert counts, and a look through its config directory turned up the opening. There are A100 configs for 8, 16, and 64 experts, but none for 128. Gemma-4 has 128, so it falls back to an untuned kernel. The tuned config for this shape simply did not exist.
So we built it. vLLM’s own benchmark_moe.py --tune swept 1920 tile configurations for the E=128, N=704 shape, after a one-line patch to its parameter reader (which assumed Mixtral’s
num_local_experts and threw on Gemma-4’s naming). The best config went into vLLM’s directory and the
benchmark re-ran.
It changed nothing: 118.2 versus 119.7 tok/s, 0.99×, dead inside the noise.
That is not a failed experiment, it is the roofline proving itself. A single-stream expert GEMM is a memory-bound GEMV, and no tile schedule beats the HBM wall when the wall is the bottleneck. Kernel tuning is a throughput lever for the compute-bound regime, not a latency lever for batch 1. The measurement confirmed the byte analysis had been right about where not to look, which reclassified the whole detour as a dead-end and pointed the search at the only remaining family: fewer passes.
What the tuned kernel did at concurrency 8
At concurrency 8 the tuned kernel was actually slightly worse, 431.4 versus 458.2 tok/s, with TTFT
rising from 267 to 362 ms. The config had been tuned for the M=1 (single-request) shape and was now
misapplied to the larger batched matmul, hurting the compute-bound prefill. A config tuned for the
wrong operating point is worse than the default. And even at 8 concurrent requests this model stays
memory-bound, because only 3.8B parameters are active per token; the compute-bound regime where kernel
tuning would pay needs far higher concurrency than any single interactive stream produces. The lesson
that survived: tune for the operating point you actually serve.
Lever 2: fewer passes (native MTP)
The second family removes whole weight-read passes rather than shrinking each one. Instead of one forward pass per token, a small drafter proposes several tokens at once, and the full model verifies them in a single pass, keeping the ones it would have produced anyway. When most proposals are accepted, one expensive weight read gets amortized across several tokens.
Gemma-4 ships its own drafter for exactly this: a 430M Multi-Token-Prediction head, trained alongside the model. The best possible drafter is one fit to the target’s own next-token distribution, and a head trained jointly with the model is as fit as it gets. Building it was not the problem. The problem was that vLLM refused to run it.
Every released vLLM build silently disabled the native drafter. Turning it on crashed the server during
the torch.compile trace with a shape mismatch: a and b must have same reduction dim, got [s47,3840]×[5632,1024]. Two attempts to work around it failed, including one against a “nightly” wheel
that turned out to be an older dev line (v0.23.1rc1.dev) carrying the identical bug. The fix had to
come from the root cause, not a workaround.
The one-line shape bug that disabled the model's own drafter
The defect was not in Gemma-4’s MTP code, which is correct by design. A recent change, PR #43957, had added an embedding-width guard to vLLM’s speculative-decoding setup: if the draft and target embedding widths differ, keep them separate. That guard is right for EAGLE-style drafters, which carry their own embeddings. It is wrong for MTP.
Gemma-4’s MTP head has a pre_projection sized for twice the backbone hidden width (5632 = 2 × 2816)
and expects to share the target’s 2816-wide embedding. With sharing disabled, the draft fell back to its
own 1024-wide embedding, so the concatenation produced 1024 + 2816 = 3840, which does not match the 5632
the projection demands. That is the [s47,3840]×[5632,1024] crash, surfacing during the compile trace.
The fix restricts the guard to EAGLE drafts (those with their own embedding tokens), so MTP shares unconditionally. One line. It then runs under full CUDA graphs with no eager fallback.
The benchmark we got wrong
Then the first real MTP measurement came back wrong, and nearly shipped as a conclusion.
Patched and running under graphs, MTP measured at 102.3 tok/s. A net loss. 0.85×, slower than doing nothing.
The first explanation was plausible and false. Acceptance was around 39%, and at that rate the drafter cannot pay for itself: each step still runs the 430M head and then verifies several positions, and on a GPU as fast as the A100 (2 TB/s) the target step is already cheap, so the drafter tax eats the token benefit. The tidy story that followed almost wrote itself: MTP’s advertised 2–3× must be a low-bandwidth-GPU phenomenon, real only where the target step is slow enough that the drafter is nearly free. It even had supporting evidence, since the numbers usually quoted for MTP came from a device running at a fraction of the A100’s bandwidth.
That story was a hardware ceiling, and it was wrong. What re-opened it was re-deriving the benchmark instead of the explanation. The harness had driven the model with random-token prompts at temperature 1.0. Both halves of that are poison for a drafter. A drafter predicts the target’s next token, and the continuation of random noise is near-uniform, so there is nothing to predict; temperature 1.0 then piles full sampling noise on top. The 39% acceptance had not measured a property of MTP or of the A100. It had measured the harness.
Re-run on coherent prose (a bundled sonnet workload) at a realistic temperature, acceptance jumped to about 87%, and the loss became the win. The collapse to 39% was a property of the input, not the method. The artifact of the bad run is kept in the repo on purpose, as a reminder that a benchmark which looks anomalous is a bug in the benchmark until proven otherwise, and that the moment to suspect the harness is exactly when a result confirms a tidy story.
Why random tokens at temperature 1.0 are the worst case for any drafter
A drafter proposes the target model’s likely next tokens. Its accuracy depends entirely on the context
being predictable. Feed it random-token context and the target’s own next-token distribution there is
near-uniform, so no drafter, however good, can guess correctly. Temperature 1.0 (inherited from the
model’s generation_config.json) then samples with full entropy, rejecting even reasonable proposals.
Content-independent levers (CUDA graphs, quantization) were unaffected by the random prompts, which is
why the same harness gave valid FP8 numbers and invalid MTP numbers.
The corrected harness used coherent Shakespeare (sonnet input 512, output 256, prefix 128) and dropped
the temperature-1.0 override with --generation-config vllm and --temperature 0 for the greedy
measurement. One more check followed before quoting anything: the first coherent result, 1.31× over
FP8, came with 94% acceptance and full 256-token generations, which smelled like greedy degeneration on
sonnet text. Re-measuring at temperature 0.7 held acceptance near 87% and the speedup at 1.48× over FP8,
proving the win was not a greedy-decoding artifact.
The drafter that looked better on paper
If the model’s own head reaches 87%, does a purpose-built external drafter do better? On paper EAGLE-3 is the stronger, more sophisticated choice, and going in it was the presumed winner.
The same corrected coherent harness measured EAGLE-3 at about 30% acceptance, 129.3 tok/s, 0.87× against FP8. A wash. A second candidate, DFlash, would not run at all: it does not support Gemma-4’s interleaved sliding and global attention (vLLM #40898).
The lesson is the one the byte budget could not supply: a drafter’s value is fit to the target model’s own next-token distribution, not raw sophistication. The model’s native head, trained jointly, fits at 87%. A more elaborate external checkpoint fits at 30% and nets nothing. This is also why the earlier random-prompt run had made EAGLE-3 look catastrophic (0% acceptance) right beside MTP, and why the re-measurement mattered so much: it separated the two. MTP’s loss was an artifact of the workload; EAGLE-3’s was real, and it came down to fit. Fit to the target, not drafter sophistication, is what picks the winner, and it is the first thing to measure on the next model.
The H20-versus-A100 mislabel behind the expected 1.9x
The expectation that EAGLE-3 would win came partly from an often-quoted result: “EAGLE-3, 1.9× on Qwen3-30B-A3B, A100.” On inspection that number was measured on an H20, a chip far more favorable to speculative decoding than the A100, not an A100 at all. The closest genuine A100-class MoE comparison lands near 1.3–1.7× for general chat. Correcting the chip label mattered: it was the difference between an expected 1.9× and the 0.87× actually measured, and it is why the native head, not the “better” external drafter, is the one that shipped.
The ladder, every rung
Each rung changes exactly one lever against vLLM’s own default, on the same GPU and model, through the same measurement path.
FP8 (fewer bytes) and MTP (fewer passes) touch independent parts of the bottleneck, so they stack. The combination is what clears 2×, and it holds under realistic sampling rather than living only at greedy:
| Config | Greedy | Temp 0.7 | Accept. |
|---|---|---|---|
| BF16 default | 127.4 | n/a | n/a |
| FP8 weights | 149.5 | 145.8 | n/a |
| BF16 + MTP | 195.1 | 176.8 | ~90% |
| FP8 + MTP | 240.4 | 215.5 | ~87% |
FP8 and MTP are independent levers and stack. FP8 + MTP greedy is 1.89× over the BF16 default and 1.61× over FP8 alone; at temp 0.7 it holds 1.48× over FP8, so the win is not a greedy-sampling artifact. Pushing num_speculative_tokens to 3 raises mean accepted length to ~4 and reaches 297.2 tok/s, the headline 2.34×.
The win grows with reply length
Speculative decoding removes whole weight-read passes, and like a fixed prefill cost that saving
amortizes over a longer generation. The headline 2.34× is measured at num_speculative_tokens=3, where
mean accepted length reaches about 4 tokens per step on coherent text. The per-token speedup starts
modest on a one-word reply and climbs toward a steady-state ceiling of 2.45× (the full 7.66 / 3.13
ratio) as the reply lengthens:
What we learned, and where this goes
Three speedups came out of this work, all measured, all real, and it matters which is which.
- 2.34× (297.2 tok/s) is FP8 + MTP at
num_speculative_tokens=3on coherent prose, mean accepted length about 4, TPOT 3.13 ms. This is the headline and the configuration wired into the live demo at the top of the page. - 1.89× (240.4 tok/s) is the same two levers at a lower speculation depth, same hardware and workload, less aggressive drafting. Both numbers are real; the difference is a knob, not a contradiction.
- 1.3–1.5× is what varied production traffic returns today, and it names the next lever rather than a limit. The speedup rides on acceptance, and the headline 87% came from structurally predictable prose, so real chat, code, and mixed traffic accept less. The fit principle points straight at the fix. A drafter fit to the production distribution, instead of to generic prose, raises acceptance, and every point of acceptance is another slice of a token per pass. The number worth chasing is acceptance on your own traffic.
None of this trades quality for speed. Speculative decoding uses rejection sampling: a proposed token is kept only if it passes the exact distribution the full model would have sampled from, and any rejected proposal falls back to a real model step. The output distribution is identical to ordinary decoding. The speedup is free of any quality cost by construction, which is the property that makes it worth shipping at all.
The path here was not a clever kernel. It was a byte budget that named the only two levers worth trying, a quantization win bounded exactly where the budget said it would be, a tuned kernel that confirmed the wall by doing nothing, and a one-line patch that switched on a fast path the model already carried. The largest trap along the way was a benchmark that measured its own harness and told a convincing story about the hardware. Re-deriving the benchmark, rather than the explanation, was the whole difference between shipping 0.85× and shipping 2.34×.
The same byte budget that named these two levers already names the next ones. The A100 left byte savings
on the table that newer silicon reclaims. FP8 KV cache, which needs sm_89, halves the KV reads the
moment the model moves to Hopper or Ada, and native FP8 there covers the experts that Ampere’s Marlin
path accelerated least, pushing more of that 38.8% under the halving. Both are pure bandwidth wins, the
only kind that moves a memory-bound step. On production traffic the second lever has the same headroom,
since acceptance is the dial and a drafter fit to the real workload turns it. Both next levers sit on
the roofline this analysis already drew, waiting on the next GPU and the next distribution.