Paper Detail
DiffusionGemma Team, Adrien Ali Taïga, James Assiene, Daniele Calandriello, Rahma Chaabouni, João Gante, Tamara von Glehn, Nate Keating, Chris Knutsen, Martin Kukla, Tianlin Liu, Ivan Lobov, Ofir Nabati, João Gabriel Oliveira, Nicolas Perez-Nieves, Nastasia Prutianova, Bobak Shahriari, Jean Tarbouriech, Pavel Tyletski, Çağlar Ünlü, Cindy Wu, Glenn Cameron, Jerome Connor, Sertan Girgin, Maarten Grootendorst, Alon Levkovitch, Eliya Nachmani, Omar Sanseviero, Piotr Stanczyk, Quentin Berthet, Andrew Campbell, Clément Crepy, Valentin De Bortoli, Arnaud Doucet, Romuald Elie, Alexandre Galashov, Klaus Greff, Alexis Jacq, David Ruhe, Yu-Han Wu, Sebastian Flennerhag, Brendan O'Donoghue, George Scrivener, Shantanu Thakoor
We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.
No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.
No ranking explanation is available yet.
No tags.
@misc{team2026diffusiongemma,
title = {DiffusionGemma Technical Report},
author = {DiffusionGemma Team and Adrien Ali Taïga and James Assiene and Daniele Calandriello and Rahma Chaabouni and João Gante and Tamara von Glehn and Nate Keating and Chris Knutsen and Martin Kukla and Tianlin Liu and Ivan Lobov and Ofir Nabati and João Gabriel Oliveira and Nicolas Perez-Nieves and Nastasia Prutianova and Bobak Shahriari and Jean Tarbouriech and Pavel Tyletski and Çağlar Ünlü and Cindy Wu and Glenn Cameron and Jerome Connor and Sertan Girgin and Maarten Grootendorst and Alon Levkovitch and Eliya Nachmani and Omar Sanseviero and Piotr Stanczyk and Quentin Berthet and Andrew Campbell and Clément Crepy and Valentin De Bortoli and Arnaud Doucet and Romuald Elie and Alexandre Galashov and Klaus Greff and Alexis Jacq and David Ruhe and Yu-Han Wu and Sebastian Flennerhag and Brendan O'Donoghue and George Scrivener and Shantanu Thakoor},
year = {2026},
abstract = {We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2},
url = {https://huggingface.co/papers/2608.00146},
keywords = {huggingface daily},
eprint = {2608.00146},
archiveprefix = {arXiv},
}
{}