Paper Detail

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

Kaixiang Zhao, Tianrun Yu, Shangzhe Li, Yuxiao Yang, Amanda Hughes, Weitong Zhang, Taylor W. Killian

arxiv Score 8.8

Published 2026-10-14 · First seen 2026-10-15

General AI

Abstract

On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@article{zhao2026recovering,
  title = {Recovering Reliable Supervision via Critique-Guided On-Policy Distillation},
  author = {Kaixiang Zhao and Tianrun Yu and Shangzhe Li and Yuxiao Yang and Amanda Hughes and Weitong Zhang and Taylor W. Killian},
  year = {2026},
  abstract = {On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the st},
  url = {https://arxiv.org/abs/2610.20572},
  keywords = {cs.LG, cs.AI},
  eprint = {2610.20572},
  archiveprefix = {arXiv},
}

Metadata

{}