Topic
mixture of experts
2 posts tagged mixture of experts.
inference
2.34× faster LLM decoding: FP8 weights and native speculative decoding on Gemma-4
Optimizing single-stream inference for Gemma-4-26B on an A100 GPU using FP8 quantization and native Multi-Token-Prediction speculative decoding.
llmspeculative decodingquantizationmixture of experts
2.34×faster Gemma-4 decoding, same output127 tok/s ▸ 297 tok/s
systems
Fixing ollama's MoE memory estimate — and a 3× kernel speedup that wasn't
Auditing Ollama and llama.cpp GPU memory estimation and quantized kernel execution. Shipped PR #17201 fixes a -20.6% VRAM under-estimation in MoE models. Includes analysis of MMQ kernel tile configurations and DeepSeek MLA execution paths.
mixture of expertscudamemory
−6.0%MoE VRAM error, from −20.6% low · shipped as PR #17201