Paper Detail

Distilled Reinforcement Learning for LLM Post-training

Chen Wang, Zhaochun Li, Jionghao Bai, Yining Zhang, Hexuan Deng, Ge Lan, Yue Wang

Browse

Workflow Queues

huggingface Score 14.0

Published 2026-07-19 · First seen 2026-07-21

General AI

Open paper source

Abstract

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

Similar Papers

Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning 2026-06-23 · Score 11.0 General AI Value Mirror Descent for Reinforcement Learning 2026-04-07 · Score 3.3 General AI The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning 2026-06-28 · Score 12.5 General AI Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents 2026-06-24 · Score 18.3 General AI Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces 2026-05-04 · Score 16.3 General AI

BibTeX

@misc{wang2026distilled,
  title = {Distilled Reinforcement Learning for LLM Post-training},
  author = {Chen Wang and Zhaochun Li and Jionghao Bai and Yining Zhang and Hexuan Deng and Ge Lan and Yue Wang},
  year = {2026},
  abstract = {Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new},
  url = {https://huggingface.co/papers/2607.17247},
  keywords = {code available, huggingface daily},
  eprint = {2607.17247},
  archiveprefix = {arXiv},
}

Metadata

{}