Surveys
On-Policy Self-Distillation for Large Language Models: A Survey
A survey of 478 papers on on-policy self-distillation, where a language model learns from itself given privileged context. It covers what the teacher sees, signal shaping, teacher dynamics, coupling with RL, the debates over when OPSD helps and what privileged information adds, and applications from agents to diffusion models.