For single-stream interactive decoding, google/gemma-4-26B-A4B-it runs at 127 tokens/sec on an A100 GPU using vLLM’s default settings. By quantizing model weights to FP8An 8-bit floating-point number format for weights. Half the bytes of BF16, so each pass reads half the weight traffic. and enabling Gemma-4’s native speculative-decodingA fast drafter proposes several tokens at once; the full model verifies them in one pass and keeps the ones it would have produced anyway, so a single weight read can cover several tokens. head, decoding throughput increases to 297 tokens/sec (2.34× faster) without altering the output distribution.
The 2.34× throughput gain corresponds to a reduction in per-token latency from 7.66 ms down to 3.13 ms. On longer generation sequences, per-token latency approaches a 2.45× speedup ceiling.
The interactive race below compares real-time token output at both baseline and optimized latencies:
This performance gain stems directly from addressing memory bandwidth bottlenecks inherent to single-request LLM decoding. The analysis below covers the memory budget breakdown, the quantization and speculative decoding implementations, kernel tuning results, and benchmarking considerations.
Memory Bandwidth Bottlenecks in Batch-1 Decoding
In single-user interactive streaming, latency is measured as Time Per Output Token (TPOTTime per output token: the wall-clock gap between one streamed token and the next. The latency a single reader actually feels.). Unlike high-concurrency throughput benchmarks that amortize weight transfers across large batch sizes, single-request decoding executes at batch size 1.
The workload uses google/gemma-4-26B-A4B-it, a fine-grained mixture of expertsA model whose feed-forward blocks are split into many experts; a router sends each token to a small subset, so total parameters far exceed the few active per token. (MoE) architecture with 25.2B total parameters and 3.8B active parameters per token (128 total experts, top-8 routed + 1 shared expert, 30 layers). Benchmarks were conducted on an A100-80GB SXM GPU running vLLM 0.25.
Architecture details and quantization constraints
Gemma-4-26B-A4B contains 25.2B total parameters with 3.8B active per token. Each layer routes to 8 of 128 experts plus one shared expert. Expert feed-forward network (FFN) width is N=704, and hidden dimension is 2816 across 30 layers. Attention uses a 5:1 ratio of sliding-window (25 layers) to global attention (5 layers). Vocabulary size is 262,144 with tied embedding and lm_head weights. A 430M Multi-Token-Prediction (MTP) draft head is included alongside the base weights.
Because expert intermediate width is relatively narrow (N=704), 4-bit group quantization introduces noticeable accuracy degradation. As a result, 8-bit formats (FP8 or INT8) are recommended for this model structure.
Roofline Analysis: Compute vs. Memory Constraints
Evaluating single-token generation under a roofline model clarifies whether decode steps are compute-bound or memory-bandwidth-bound. At batch size 1, matrix multiplications reduce to matrix-vector products (GEMVA matrix-vector product. Batch-1 decode multiplies each weight matrix by one activation vector, reading every weight once for a single multiply-add.).
A GEMV performs approximately 2 FLOPs for each 2-byte parameter loaded (in BF16), giving an arithmetic intensityUseful work per byte moved, in FLOPs per byte. A matrix times a single vector does about 1 FLOP per weight byte it reads. of ~1 FLOP/byte.
On an NVIDIA A100 SXM (2.04 TB/s HBM bandwidth, 312 TFLOP/s BF16 tensor core performance), the roofline ridge pointThe arithmetic intensity where a kernel stops being bandwidth-bound and becomes compute-bound: peak FLOP/s divided by peak bytes/s. About 153 FLOP/byte for the A100 in BF16. sits at ~153 FLOPs/byte. Operating at ~1 FLOP/byte puts single-stream decoding deep within the memory-bandwidth-boundLimited by how fast weights stream out of HBM, not by how fast the cores multiply. Latency tracks bytes moved per token. regime.
As a result, GPU compute cores spend most of their time waiting for weight data to stream from High-Bandwidth Memory (HBMHigh-bandwidth memory, the GPU DRAM that holds the weights. The A100 reads it at about 2.04 TB/s, which sets the decode floor.). Optimizations must focus on either:
- Reducing bytes read per pass (e.g., weight quantization).
- Reducing passes required per token (e.g., speculative decoding).
Derivation of arithmetic intensity for batch-1 decode
A matrix-vector multiplication of shape (M=1, K, N) reads K × N weight elements and performs 2 × K × N floating-point operations. For 16-bit precision (2 bytes per weight), loading 2 × K × N bytes yields an arithmetic intensity of (2 × K × N FLOPs) / (2 × K × N bytes) = 1 FLOP/byte.
With the A100’s ridge point at 153 FLOPs/byte, execution is constrained by HBM read rates rather than raw ALUs. Increasing batch size increases reuse of loaded weights across multiple prompt sequences, shifting execution toward the compute-bound regime. However, for a single isolated stream, batch size remains 1.
Calculating memory traffic per token from model configuration shapes yields 7.36 GB of weight data loaded per decoding step in BF16:
- experts38.8% · 2.86 GB
- attention26.3% · 1.93 GB
- lm_head20.1% · 1.48 GB
- dense MLP14.5% · 1.07 GB
- router0.3% · 0.02 GB
At 2.04 TB/s peak HBM bandwidth, transferring 7.36 GB sets a theoretical minimal step duration of ~3.6 ms (~278 tok/s max).
Detailed breakdown of memory reads per token
Exact weight memory reads per token in BF16 (2 bytes/param):
- Routed Experts: 2.86 GB (38.8%) — 8 active experts out of 128 across 30 layers, reading gate, up, and down projections (1.43B parameters).
- Attention Projections: 1.93 GB (26.3%) — Q, K, V, and O projections across 30 layers (966M parameters).
lm_head: 1.48 GB (20.1%) — Tied output embedding layer of shape2816 × 262,144(738M parameters).- Shared MLP: 1.07 GB (14.5%) — Always-on feed-forward block across 30 layers (535M parameters).
- Router: 0.02 GB (0.3%) — Expert routing matrices.
Note: This calculation excludes KV-cache reads and kernel launch overheads, explaining why baseline runtime (7.66 ms/token) sits above the theoretical 3.6 ms floor.
vLLM utilizes CUDA graphsA captured, replayable recording of a step's GPU kernels. One replay launches the whole step at once, erasing the per-kernel Python launch overhead. by default, which eliminates host-side Python invocation overhead. Disabling CUDA graphs drops execution throughput from 127.4 tok/s down to 13.5 tok/s (a 9.4× slowdown), due to kernel launch overhead across thousands of tiny MoE operations per step.
Lever 1: FP8 Weight Quantization
Quantizing model weights from 16-bit (BF16) to 8-bit (FP8) reduces weight memory bandwidth demands by 50%.
In vLLM, passing --quantization fp8 quantizes model weights during load time and dispatches matrix multiplications to FP8-MarlinThe vLLM kernel that stores weights in FP8 and dequantizes them to BF16 in registers for the matmul. On Ampere it saves HBM bytes only, since the A100 has no FP8 tensor cores. kernels.
Applying FP8 weight quantization increases throughput from 127.4 to 149.5 tok/s (+17% gain).
Hardware constraints on FP8 execution (Ampere vs. Hopper)
NVIDIA Ampere GPUs (A100, sm_80) do not contain native FP8 Tensor Cores; native FP8 hardware acceleration was introduced in Hopper (sm_90) and Ada Lovelace (sm_89).
On Ampere, FP8-Marlin loads 8-bit weight vectors from HBM into registers and unpacks them to BF16 prior to execution on BF16 Tensor Cores. Consequently, FP8-Marlin saves HBM memory bandwidth during weight loads, but does not increase compute throughput. Furthermore, FP8 quantization does not accelerate all layers equally on Ampere, which accounts for the measured +17% throughput increase relative to theoretical limits.
Attempting to quantize the KV cacheThe stored keys and values from earlier tokens that attention reads each step, so past context is never recomputed. to FP8 on Ampere resulted in runtime errors, as FP8 KV-cache execution in vLLM requires sm_89 or newer architectures.
Evaluating Custom MoE Kernel Tuning
Before configuring speculative decoding, we tested whether custom GEMV kernel tuning could improve MoE execution speed. vLLM includes tuned fused-MoE configurations for 8, 16, and 64 expert setups, but defaults to untuned fallback kernels for Gemma-4’s 128-expert layout.
Using vLLM’s benchmark_moe.py --tune tool, we benchmarked 1,920 candidate tile configurations for the E=128, N=704 shape and generated a tuned configuration file.
Comparing performance before and after applying the tuned configuration yielded no significant change (118.2 tok/s vs. 119.7 tok/s, within noise margins). Because batch-1 decode steps are limited by HBM bandwidth rather than tile execution efficiency, kernel tuning provides minimal benefit in single-stream decoding.
Impact of MoE kernel tuning under concurrency
Evaluating the tuned MoE configuration at concurrency 8 produced slightly lower performance (431.4 tok/s vs 458.2 tok/s default). The configuration was tuned specifically for batch size 1 (M=1), resulting in sub-optimal tile choices when applied to larger batch sizes during prefill phases.
Lever 2: Native Multi-Token Prediction (MTP)
Speculative decoding reduces total memory passes by generating multiple draft tokens per iteration and verifying them in a single forward pass of the target model.
Gemma-4 ships with an integrated 430M Multi-Token-PredictionA drafter head trained jointly with the target model to predict several of its next tokens. Because it is fit to the target's own distribution, its proposals are accepted often. (MTP) draft head. Because the MTP head was trained jointly alongside the main model weights, its output distribution matches the base model closely.
In vLLM 0.25, enabling MTP initially resulted in a shape mismatch error during CUDA graph compilation (a and b must have same reduction dim, got [s47,3840]×[5632,1024]).
Root cause and fix for vLLM MTP initialization crash
The crash was caused by an embedding guard added in PR #43957. The guard assumed draft heads maintain separate token embedding tables (as in EAGLE), whereas Gemma-4’s MTP head shares embedding weights directly with the target backbone.
Gemma-4’s MTP projection expects a concatenated hidden state of width 2 × 2816 = 5632. When separate embeddings were forcibly instantiated, the draft layer received an input width of 1024 + 2816 = 3840, causing dimension mismatches during matmul execution.
Restricting the embedding guard to EAGLE models (PR #47953) resolved the issue, allowing MTP to initialize correctly and execute within CUDA graphs.
Benchmarking Evaluation Metrics and Prompt Selection
Initial MTP benchmark runs resulted in lower throughput (102.3 tok/s, 0.80× baseline), with an observed acceptance rateThe fraction of a drafter's proposed tokens the full model keeps. Higher acceptance means more tokens per verification pass, and more speedup. of ~39%.
Analyzing the benchmark setup showed that the test harness was issuing synthetic random-token prompts at temperature 1.0. Random prompt tokens lack coherent linguistic structure, leading to flat next-token probability distributions. Under uniform distributions, draft proposals fail verification frequently.
Re-evaluating with coherent text prompts and representative sampling parameters (temperature 0.7) restored draft acceptance rates to ~87%, increasing decoding throughput to 297.2 tok/s.
Validation across temperature settings
To confirm that acceptance gains were not artifacts of greedy decoding, we tested across multiple sampling parameters:
- At temperature 0 (greedy decoding), acceptance rate reached 94%, producing 195.1 tok/s on BF16 and 297.2 tok/s on FP8 (
num_speculative_tokens=3). - At temperature 0.7, acceptance rate settled at ~87%, delivering 215.5 tok/s at
num_speculative_tokens=2(1.48× over FP8 baseline).
This confirms that speculative decoding performance benefits persist under standard non-zero sampling configurations.
Comparison with External Drafters (EAGLE-3)
We also evaluated EAGLE-3, an external speculative drafting architecture. On the same coherent evaluation dataset, EAGLE-3 achieved a ~30% acceptance rate, yielding 129.3 tok/s (0.87× relative to FP8 baseline).
Because Gemma-4’s native MTP head was trained jointly with the base model, its draft predictions align more accurately with target model logits than the standalone EAGLE-3 checkpoint.
Hardware context behind reported speculative decoding gains
Literature benchmarks for EAGLE-3 often report ~1.9× gains on GPUs such as the NVIDIA H20. The H20 features reduced compute capacity relative to HBM memory bandwidth compared to the A100. On memory-constrained cards like the H20, draft execution overhead represents a smaller fraction of step time, yielding higher relative speedup ratios than on an A100.
Benchmark Summary across Configurations
Performance gains for each optimization stage are summarized below:
Because weight quantization (reducing bytes per pass) and speculative decoding (reducing pass count) address distinct aspects of the memory bottleneck, their gains combine multiplicatively:
| Config | Greedy (tok/s) | Temp 0.7 (tok/s) | Acceptance Rate |
|---|---|---|---|
| BF16 default | 127.4 | N/A | N/A |
| FP8 weights | 149.5 | 145.8 | N/A |
| BF16 + MTP | 195.1 | 176.8 | ~90% |
| FP8 + MTP (num_spec=2) | 240.4 | 215.5 | ~87% |
| FP8 + MTP (num_spec=3) | 297.2 | N/A | ~87% |
Combining FP8 weight quantization with MTP speculative decoding yields 297.2 tok/s at greedy settings (2.34× speedup) and 215.5 tok/s at temperature 0.7.
Latency vs. Output Sequence Length
Because draft verification overhead occurs per step, speculative decoding gains increase with generated output sequence length. At num_speculative_tokens=3, per-token latency approaches a 2.45× steady-state ceiling as generation length increases:
Correctness and Verification
Speculative decoding employs rejection samplingThe rule that keeps speculative decoding exact: a proposed token is accepted only if it passes the distribution the full model would have sampled, and a rejection falls back to a real model step. during verification. Proposed tokens are accepted only when they match samples drawn from the target model’s output probability distribution. Rejected tokens trigger a standard forward pass of the main model.
As a result, speculative decoding maintains mathematical identity with unspeculated decoding outputs, introducing zero loss in generation quality.
Moving this workload to Hopper-based GPUs (e.g. H100/H200) will unlock additional speedups via native FP8 Tensor Cores and FP8 KV-cache support.
Glossary
Glossary — every term, defined
- TPOT
- Time per output token: the wall-clock gap between one streamed token and the next. For a single interactive stream it is the latency a person feels, distinct from aggregate throughput across many users.
- Roofline
- A plot of achievable
FLOP/sagainst arithmetic intensity. Performance rises with intensity until it flattens at a compute ceiling; the corner is the ridge point. - Ridge point
- The arithmetic intensity where a kernel stops being bandwidth-bound and becomes compute-bound: peak FLOP/s divided by peak bytes/s. About 153 FLOP/byte for the A100 in BF16.
- Arithmetic intensity
- Useful work per byte moved, in FLOPs per byte. A matrix times a single vector (a GEMV) does about 1 FLOP per weight byte, far below the ridge.
- GEMV
- A matrix-vector product. Batch-1 decode multiplies each weight matrix by one activation vector, reading every weight once for a single multiply-add each.
- Memory-bandwidth-bound
- Limited by how fast weights stream out of
HBM, not by how fast the cores multiply. Latency tracks bytes moved per token, so the only levers that help move fewer bytes. - HBM
- High-bandwidth memory, the GPU DRAM that holds the weights. The A100 reads it at about 2.04 TB/s, which sets the decode floor.
- Mixture of experts (MoE)
- A model whose feed-forward blocks are split into many experts; a router sends each token to a small subset. Gemma-4 routes the top 8 of 128 experts plus one shared, so 25.2B parameters yield only 3.8B active per token.
- FP8
- An 8-bit floating-point weight format, half the bytes of BF16. Weight-only FP8 halves the weight traffic each pass.
- FP8-Marlin
- The vLLM kernel that stores weights in FP8 and dequantizes them to BF16 in registers for the matmul. On Ampere it saves HBM bytes only, since the A100 has no FP8 tensor cores.
- KV cache
- The stored keys and values from earlier tokens that attention reads each step, so past context is never recomputed. Quantizing it to FP8 needs
sm_89, which Ampere lacks. - CUDA graphs
- A captured, replayable recording of a step's GPU kernels. One replay launches the whole step, erasing per-kernel Python launch overhead. On this MoE model it is worth 9.4× over eager.
- Speculative decoding
- A fast drafter proposes several tokens at once; the full model verifies them in one pass and keeps the ones it would have produced anyway. It removes whole weight-read passes without changing the output.
- Multi-Token-Prediction (MTP)
- A drafter head trained jointly with the target model to predict several of its next tokens. Because it is fit to the target's own distribution, its proposals are accepted often; Gemma-4 ships a 430M MTP head.
- Acceptance rate
- The fraction of a drafter's proposed tokens the full model keeps. Higher acceptance means more tokens per verification pass; the speedup rides directly on it.
- Rejection sampling
- The rule that keeps speculative decoding exact: a proposed token is accepted only if it passes the distribution the full model would have sampled, and a rejection falls back to a real model step. The output distribution is unchanged.