UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models

Abstract

Uniform Discrete Diffusion Model (UDM) has recently emerged as a promising paradigm for discrete generative modeling; however, its integration with reinforcement learning remains largely unexplored. We observe that naively applying GRPO to UDM leads to training instability and marginal performance gains. To address this, we propose \Ours, the first framework to integrate UDM with RL. Our method is guided by two key insights: (i) treating the final clean sample as the action provides more accurate and stable optimization signals; and (ii) reconstructing trajectories via the diffusion forward process better aligns probability paths with the pretraining distribution. Additionally, we introduce two strategies, Reduced-Step and CFG-Free, to further improve training efficiency. \Ours significantly improves base model performance across multiple T2I tasks. Notably, GenEval accuracy improves from 69% to 96% and PickScore increases from 20.46 to 23.81, achieving state-of-the-art performance in both continuous and discrete settings. On the OCR benchmark, accuracy rises from 8% to 57%, further validating the generalization ability of our method. Code is available at https://github.com/Yovecent/UDM-GRPO{https://github.com/Yovecent/UDM-GRPO}.

Workflow Status

Review status

pending

Role

unreviewed

Read priority

now

Vote

Not set.

Saved

no

Collections

Not filed yet.

Next action

Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

BibTeX

@misc{wang2026udm,
  title = {UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models},
  author = {Jiaqi Wang and Haoge Deng and Ting Pan and Yang Liu and Chengyuan Wang and Fan Zhang and Yonggang Qi and Xinlong Wang},
  year = {2026},
  abstract = {Uniform Discrete Diffusion Model (UDM) has recently emerged as a promising paradigm for discrete generative modeling; however, its integration with reinforcement learning remains largely unexplored. We observe that naively applying GRPO to UDM leads to training instability and marginal performance gains. To address this, we propose \textbackslash{}Ours, the first framework to integrate UDM with RL. Our method is guided by two key insights: (i) treating the final clean sample as the action provides more accurat},
  url = {https://huggingface.co/papers/2604.18518},
  keywords = {Uniform Discrete Diffusion Model, reinforcement learning, GRPO, diffusion forward process, trajectory reconstruction, Reduced-Step, CFG-Free, text-to-image tasks, OCR benchmark, GenEval, PickScore, code available, huggingface daily},
  eprint = {2604.18518},
  archiveprefix = {arXiv},
}

Metadata

{}