inferenceJul 22, 2026 · 17 min
2.34× faster LLM decoding: FP8 weights and native speculative decoding on Gemma-4
Optimizing single-stream inference for Gemma-4-26B on an A100 GPU using FP8 quantization and native Multi-Token-Prediction speculative decoding.
llmspeculative decodingquantizationmixture of experts
2.34×faster Gemma-4 decoding, same output127 tok/s ▸ 297 tok/s
