Surveys

On-Policy Self-Distillation for Large Language Models: A Survey

Abstract

On-policy self-distillation (OPSD) trains a language model on its own samples against a teacher that is the same model given information the student does not see: a reference solution, environment feedback, a rubric, a better view of the input, or a consensus of its own attempts. The resulting signal is dense, token-level and on-policy, and it needs neither a separate teacher model nor a group of rollouts per prompt. Since January 2026 the idea has grown from a handful of reports into several hundred papers. This survey organizes 478 papers collected up to 3 November 2026 along the components of the OPSD pipeline: what the self-teacher sees, how the distillation signal is shaped, how the teacher evolves, and how self-distillation is coupled with reinforcement learning. A second axis covers applications in agents, multimodal models, generative models, continual learning, safety and efficient reasoning. The survey draws on 144 papers read in full. It finds that the privileged context determines where OPSD applies. It also finds that controlled studies attribute a large share of the teacher's value to context-induced behavior rather than to answer-specific knowledge. Gains are robust for non-thinking students with a large teacher–student gap and often reverse for thinking models evaluated at long budgets. The dominant recent designs therefore let a verifier set the direction of the update and use the self-teacher only for token-level magnitude or for a few targeted steps. We close with open problems in measuring privileged information, preventing collapse, scaling beyond 14B models and auditing the behavioral side effects of self-distillation.

1. Introduction

Post-training of large language models relies on three kinds of learning signal, and each has a familiar weakness. Supervised fine-tuning imitates fixed demonstrations, so it is off-policy and can overwrite capabilities the model already has. Reinforcement learning with verifiable rewards (RLVR) trains on the model's own samples, but it receives one scalar per sequence and learns nothing from a group of rollouts that are all right or all wrong. On-policy distillation (OPD) combines the two strengths, training on the student's own samples with a dense per-token target, but it requires a separate, usually larger teacher model. On-policy self-distillation removes that last requirement. The same weights act as teacher and student, and the teacher becomes better than the student only because it is given privileged information cc that the student will not have at test time. A reference solution, an execution error, a rubric or a zoomed-in crop of an image can turn the model into its own teacher .

The idea spread quickly. In the corpus assembled for this survey, 7 relevant papers appeared in 2025, 31 more between January and April 2026, and 440 between May and the start of November 2026. The literature now spans math and science reasoning, multi-turn agents, vision and video models, diffusion generators, knowledge editing, safety alignment and speech recognition. Its growth has also produced controlled analyses that question the assumptions of the founding papers: whether the teacher's advantage comes from the content of the privileged context , whether OPSD helps thinking models at all , and whether it really forgets less than RL . A survey is timely because the method papers and the analysis papers have so far developed largely in parallel.

Four earlier overviews exist. They cover on-policy distillation broadly , give a brief early overview of OPSD , derive OPD variants from one formula or critically review the collapse problem in math reasoning . Section 2.3 compares them with this survey. Relative to them, this survey (i) is organized around the OPSD pipeline instead of loss formulas, (ii) covers the full application range rather than math alone, and (iii) puts the analysis literature on an equal footing with method papers, devoting a chapter to the foundational questions it raises.

Scope. We include methods in which the teacher distribution is produced by the student's own weights (frozen, moving, trained or architecturally modified) under a context or input the student does not see, and in which supervision is applied on the student's own samples. We exclude on-policy distillation from a separate, larger teacher, except where a paper's analysis bears directly on self-distillation, and we exclude off-policy self-training without a token-level teacher signal. Appendix A gives the retrieval routes, the cutoff and the screening procedure.

Organization. Section 2 introduces the shared notation, a timeline and the related surveys. Section 3 presents the taxonomy. Sections 4–7 cover the four pipeline components: the privileged context, signal shaping, teacher dynamics and coupling with RL. Section 8 examines six foundational questions. Section 9 surveys applications, Section 10 collects benchmarks and systems, Section 11 discusses open problems and Section 12 concludes.

mermaid
flowchart LR X[Prompt x] --> S[Student samples y from p] X --> T[Same weights with privileged context c] S --> T T --> Q[Teacher distribution q on the student's prefixes] S --> P[Student distribution p] Q --> L[Token loss: weighted divergence between p and q] P --> L V[Verifier or reward, optional] --> L L --> U[Update student weights] U -. frozen, EMA, trained or lookahead .-> T

Figure 1. The OPSD loop. The student samples on its own; the same weights, given privileged context cc, score the student's prefixes; a per-token divergence, optionally combined with a verifier signal, updates the student; and a teacher-update rule decides how the teacher's weights follow. Sections 4–7 each address one component of this loop.

2. Preliminaries

2.1 Setting and notation

Let xx be a prompt and πθ\pi_\theta the policy being trained. The student samples a response y∼πθ(⋅∣x)y\sim\pi_\theta(\cdot\mid x) and, at position tt, has next-token distribution pt=πθ(⋅∣x,y<t)p_t=\pi_\theta(\cdot\mid x,y_{<t}). The self-teacher is a copy πˉ\bar\pi of the same model, evaluated on the same prefix but with privileged context cc:

qt=πˉ(⋅∣x,c,y<t),πˉ∈{πθ0, EMA(πθ), sg(πθ), …}.q_t=\bar\pi(\cdot\mid x,c,y_{<t}),\qquad \bar\pi\in\{\pi_{\theta_0},\ \mathrm{EMA}(\pi_\theta),\ \mathrm{sg}(\pi_\theta),\ \ldots\}.

The generic OPSD objective is a weighted sum of per-token divergences,

L(θ)=Ex Ey∼πθ(⋅∣x)[∑twt D(pt, sg(qt))],\mathcal{L}(\theta)=\mathbb{E}_{x}\,\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}\Big[\sum_{t} w_t\,D\big(p_t,\ \mathrm{sg}(q_t)\big)\Big],

where sg\mathrm{sg} denotes stop-gradient, wtw_t is a token weight (uniform 1/∣y∣1/|y| in the vanilla form), and DD is a divergence: reverse KL KL(pt∥qt)\mathrm{KL}(p_t\Vert q_t), forward KL, or a Jensen–Shannon interpolation. When DD is reverse KL and only the sampled token is used, the per-token reward is the log-ratio rt=log⁡qt(yt)−log⁡pt(yt)r_t=\log q_t(y_t)-\log p_t(y_t). SDPO showed that the gradient of the reverse-KL objective is a policy gradient with exactly this advantage , which is what connects OPSD to RL. A verifier reward r(x,y)∈{0,1}r(x,y)\in\{0,1\} and a group-relative advantage AA appear when OPSD is combined with RLVR (Chapter 7).

SymbolMeaning
xx, yy, y<ty_{<t}prompt, student-sampled response, prefix before token tt
ccprivileged context seen only by the teacher (solution, feedback, rubric, view, consensus)
πθ\pi_\theta, πˉ\bar\pistudent policy; teacher copy of the same model (frozen, EMA, trained)
ptp_t, qtq_tstudent and teacher next-token distributions on the student's prefix
DD, wtw_tdivergence and per-token weight in the distillation loss
rtr_tsampled-token log-ratio log⁡qt(yt)−log⁡pt(yt)\log q_t(y_t)-\log p_t(y_t)
r(x,y)r(x,y), AAverifier reward and RL advantage, when OPSD is combined with RLVR

Table 1. Shared notation used throughout the survey. Individual papers use different symbols; we translate their objectives into this notation.

OPSD differs from its neighbors along two axes. Unlike context distillation in its original off-policy form, supervision is applied on the student's own samples. Unlike OPD with a separate teacher, the teacher's advantage comes from cc, not from more parameters. The second difference has a practical consequence and a conceptual one. In practice, the teacher costs one extra forward pass over a longer context of the same model, and no second model needs to be served . Conceptually, the student can only learn what the model can already express when given cc, which bounds the achievable gain (Section 8.5).

2.2 Development of the field

The founding reports appeared within a few days of each other in January 2026. The Self-Distilled Reasoner (OPSD) conditioned a frozen teacher on reference solutions for math reasoning . SDFT used demonstrations for continual learning . SDPO used environment feedback in a policy-gradient framing . February added privileged-information distillation for agents , RL from text feedback and on-policy context distillation . The volume grew sharply from May 2026, when variants of every pipeline component, the first systematic diagnostics and the first applications to vision, diffusion and agents appeared within weeks of each other.

mermaid
timeline title Representative OPSD milestones in 2026 January : OPSD with reference solutions [1] : SDFT for continual learning [3] : SDPO with rich feedback [2] February-March : Privileged-information distillation for agents [14] : On-policy context distillation [13] : HDPO hybrid with RL [16] April : Self-Distilled RLVR [17] : SD-Zero self-revision [18] : Skill-SD for agents [19] May : Vision-OPD [20] : Anti-self-distillation [21] : D-OPSD for diffusion [22] June-July : Thinking-model failures [6] : Label-free CANON [23] : dOPSD for diffusion LMs [24] August-November : Mismatched-reference controls [4] : Diagnostic sweeps [7] : Asynchronous OPSD [25]

Figure 2. Timeline of representative OPSD works by arXiv submission month, chosen to mark the first appearance of each major line rather than the most cited works.

Period (arXiv ID month)Papers in corpusCharacter of the work
June–December 20257precursors: adaptive guidance, self-supervised RL, self-generated preferences and training-aware speculative decoding
January–April 202631founding recipes, first RL hybrids and agent adaptations
May–November 2026440rapid diversification of components, applications and analyses

Table 2. Distribution of the 478 corpus papers by arXiv identifier month (a proxy for first submission). November 2026 is partial (first three days).

Table 3 compares the four earlier overviews with this survey. The OPD survey decomposes on-policy distillation into the visited state distribution, the teacher's view and the gradient used, and treats self-distillation as one case of a context-conditioned teacher. The brief overview compares OPSD with OPD in a qualitative table and an eight-way design taxonomy, without experiments. The formula-driven survey derives nine variables of the feedback-to-update path and proposes untested variants. The critical review regroups six months of follow-up work in math reasoning under one symptom, collapse, and three levers.

SurveyCutoff (approx.)ScopeOrganizing axisApplications covered
OPD survey April 2026all on-policy distillationstate distribution, teacher view, gradientpartial
Brief overview May 2026OPSDeight design dimensionsnone
Formula-driven survey June 2026all on-policy distillationnine formula variablesnone
Critical review August 2026OPSD for math reasoningone symptom, three leversexcluded
This surveyNovember 2026OPSD, 478 paperspipeline components plus applicationsagents, multimodal, generative, continual, safety, efficiency, other

Table 3. Related surveys. Cutoffs are inferred from arXiv identifier months and the works each overview discusses.

Takeaways

  • OPSD is on-policy distillation in which the teacher is the same model with privileged context. Its reverse-KL form is a policy gradient with a token-level log-ratio advantage.
  • The field formed in January 2026, and 92% of the corpus (440 of 478 papers) appeared after April 2026.
  • Earlier overviews are either broader (all on-policy distillation) or narrower (math only, or formula-level). None covers the application range or the analysis literature as a whole.

3. A Taxonomy of On-Policy Self-Distillation

We organize the literature along two axes. The mechanism axis follows the components of the OPSD loop in Figure 1. It asks what the teacher sees, how the signal is shaped, how the teacher evolves and how self-distillation is combined with RL, and it adds two cross-cutting categories: analyses and theory, and training systems. The application axis records the domain a paper targets. Papers can carry several labels on each axis, and most do: a typical agent paper both constructs a new privileged context and couples it with GRPO. Figure 3 shows the top levels with representative works, and Table 4 gives the size of each category in the corpus.

mermaid
mindmap root((On-policy self-distillation)) What the teacher sees · Ch. 4 Solutions and demonstrations · 1, 3, 30 Feedback and hindsight · 2, 15, 31 Rubrics skills memory · 32, 33, 34 Label-free privilege · 23, 35, 36 Signal shaping · Ch. 5 Token and step weights · 37, 38 Divergence and estimator · 39, 40, 21 Problem and rollout level · 41, 42 Teacher dynamics · Ch. 6 Frozen and gated refresh · 43 Trained teachers · 44, 45 Lookahead teachers · 46, 47 Coupling with RL · Ch. 7 Additive losses · 16, 18 Ratios in the advantage · 17, 48 Routing · 49, 50 Analyses and debates · Ch. 8 When it helps · 6, 51 What privilege adds · 4, 5 Collapse and forgetting · 52, 8 Applications · Ch. 9 Agents · 19, 53 Multimodal · 20, 54 Generative · 24, 22 Safety and efficiency · 55, 56

Figure 3. Taxonomy of on-policy self-distillation with representative works at the leaves.

Mechanism categoryPapersApplication categoryPapers
Teacher context (Ch. 4.1–4.3)179General reasoning119
Label-free privilege (Ch. 4.4)133Agents and tool use107
Signal shaping (Ch. 5)195Multimodal understanding73
Teacher dynamics (Ch. 6)47Continual learning and knowledge40
Coupling with RL (Ch. 7)158Alignment and safety40
Analysis and theory (Ch. 8)78Generative models31
Training systems (Ch. 10)12Reasoning efficiency27
Surveys and position papers4Code and software engineering17
Separate-teacher OPD (boundary cases)8Other domains19
Uncertain on this axis40Uncertain on this axis98

Table 4. Category sizes on both axes for the 478-paper corpus. Labels were assigned by an LLM-based multi-label classifier and are not exclusive, so columns do not sum to 478. The "label-free" count is broad: it includes papers whose privilege is self-generated even when some supervision is used elsewhere.

Three regularities in the label distribution shape the rest of the survey. First, signal shaping and RL coupling are both larger than teacher dynamics by a factor of three or more. The field has spent more effort on how to use the teacher's distribution than on which teacher to use. Second, a sixth of the corpus is analysis and theory, an unusually high share for a field less than a year old, which is why Chapter 8 is as long as the method chapters. Third, agents and multimodal models together account for more papers than general reasoning. OPSD is no longer a math-reasoning technique with a few side applications.

Takeaways

  • The mechanism axis follows the four components of the OPSD loop (context, signal, teacher dynamics, RL coupling), and an application axis records the domain.
  • Research effort concentrates on shaping and combining the teacher signal. Teacher dynamics is comparatively underexplored.
  • Analyses make up about a sixth of the corpus, and applications outside reasoning now outnumber reasoning-only work.

4. Constructing the Self-Teacher: What the Teacher Sees

Every OPSD method has to answer one design question before any loss is written: what information cc turns the frozen copy πˉ\bar\pi into a teacher whose next-token distribution qt=πˉ(⋅∣x,c,y<t)q_t=\bar\pi(\cdot\mid x,c,y_{<t}) is better than the student's ptp_t? The answer determines what data the method needs (reference solutions, a verifier, rubrics, nothing at all), what the teacher can know that the student cannot, and how much of that knowledge can be absorbed into weights that never see cc at test time. This chapter organizes the literature by the content of cc, moving from the richest and most label-dependent contexts to contexts the model manufactures for itself.

mermaid
mindmap root((Privileged context c)) Solutions and demonstrations Full reference solution · 1, 3 Partial or masked views · 57, 58, 30 Distilled guidance · 59 Purified targets · 60 Feedback and hindsight Environment or interpreter feedback · 2, 15 Self-reflection and critique · 31, 61 Repaired reasoning · 62 Peer rollouts · 63 Rubrics skills and memory Rubrics · 32 Skill cards · 33 Procedural memory · 34 Extracted experience · 13 Label-free privilege Consensus solutions · 23, 35, 64 Weakened weights · 36 Self-generated data · 65

Figure 4. Subtree of Chapter 4: privileged contexts used by on-policy self-distillation methods, with representative works (arXiv identifiers stand in for the citation numbers resolved in the text).

4.1 Reference solutions and demonstrations

The canonical form of OPSD conditions the self-teacher on a verified worked solution. The Self-Distilled Reasoner made the construction concrete for math reasoning: the same model is prompted once with the problem alone (student) and once with the problem, a reference chain-of-thought y⋆y^\star and the instruction to "solve this problem using your own approach" (teacher). The teacher emits no tokens; one prefilled forward pass over the student's rollout yields qtq_t at every position, and the student minimizes a per-token, pointwise-clipped divergence to it. Two choices in that paper became defaults for the line: the teacher is the initial policy πθ0\pi_{\theta_0}, which the authors found more stable than the moving policy, and a single 1,024-token rollout per prompt replaces the group of long rollouts needed by GRPO. With these settings the authors report average Avg@12 on AIME24/AIME25/HMMT25 rising from 37.1 to 43.4 for Qwen3-1.7B (GRPO: 37.7) and from 61.8 to 64.8 for Qwen3-8B (GRPO: 64.0), at roughly 1/128 of GRPO's sampled tokens . Figure 5 shows the construction.

Figure 5. On-policy self-distillation with a reference solution as privileged context: the student samples from the problem-only prompt, and the same weights, prompted with the reference solution, score every prefix of that sample (reproduced from the Self-Distilled Reasoner ).

Concurrently, Self-Distillation Fine-Tuning (SDFT) applied the same pattern to supervised data in general: the teacher is the model conditioned on the per-query demonstration, the student trains on its own rollouts against it with an EMA teacher, and the loss is summed over the full vocabulary. Its target was not reasoning accuracy but retention. In a knowledge-acquisition task SDFT reaches 89/100/98 strict/lenient/out-of-distribution accuracy against 80/95/80 for SFT, and in a three-task sequential setting it learns each task without the regressions SFT shows . This established a second motivation for OPSD that recurs in Chapter 8: because the targets are the student's own tokens, reweighted rather than replaced, OPSD moves the model less than SFT does.

Figure 6. The SDFT mechanism: the demonstration is placed in the teacher's context only, and the student learns from its own samples (reproduced from ).

A large follow-up literature asks how much of the reference the teacher should see. The answers converge on "less than all of it." Adaptive Teacher Exposure truncates the reasoning prefix of the reference to a fraction α\alpha (always keeping the final answer) and learns α\alpha as a training-state-conditioned control variable; it reports Avg@12 of 44.35/65.65/67.13 at 1.7B/4B/8B against 43.4/63.6/64.8 for OPSD. PAINT masks a suffix of the reference whose length depends on how much of the reference the rollout already reproduces, and interpolates teacher and student energies on a sparse set of high-entropy-ratio positions; at 8B it reports 66.9 against 64.8 for OPSD. A systematic study of granularity compiles five matched contexts, from the full solution to a named strategy, a method-independent framing, a problem category and the answer alone. At 4B and 8B an intermediate level beats the full solution (64.81 vs 63.43; 68.24 vs 66.67) while storing 17–37× fewer hint tokens, but at 1.7B the full solution still wins narrowly. Averaging the teacher over several randomly masked views lowers the sensitivity of the target to which valid reference was supplied, and Problem-Space Guidance replaces the solution with a four-field extracted record. Self-Privileged Fine-Tuning carries the idea into supervised fine-tuning: the frozen prior that weights SFT tokens is itself conditioned on privileged outcome information, which improves over SFT on math and code but not uniformly over the unconditioned prior. Across these works the explanation offered is the same: a full solution pulls the teacher toward copying a particular derivation, and that path-specific signal does not transfer to a student that never sees it.

That explanation was tested directly. Rethinking Privileged Information decomposes the teacher–student log-ratio into a "reference" direction and a "recovery" direction and finds that, at 8B on NuminaMath with a non-thinking teacher, a mismatched reference from another problem outperforms the correct one (average 67.6 vs 59.5, with paired intervals excluding zero), while on MegaScience the two differ by at most 0.3 points. Purified OPSD uses two additional frozen forward passes (reference only, question only) to keep only the question-conditioned part of the teacher's log-ratio, and reports gains across four models and two datasets (Qwen3-8B on DASD-10K: 66.0 vs 60.9 for standard OPSD). Taken together, these results suggest that the reference solution acts at least partly as a prompting intervention that changes the teacher's behavior, rather than as answer-specific knowledge that the student absorbs; Section 8.2 returns to this question.

4.2 Feedback, critique and hindsight

The second family makes the teacher wiser by showing it what happened, not what should have happened. SDPO formulated reinforcement learning with rich feedback: the environment's feedback ff on a rollout (an interpreter error, a failing test, a judge's remark) is appended to the teacher's prompt, and the student minimizes ∑tKL(pt ∥ sg πθ(⋅∣x,f,y<t))\sum_t \mathrm{KL}(p_t\,\|\,\mathrm{sg}\,\pi_\theta(\cdot\mid x,f,y_{<t})). The authors show that the gradient of this loss is a policy gradient with token-level advantage At=log⁡πθ(yt∣x,f,y<t)πθ(yt∣x,y<t)A_t=\log\frac{\pi_\theta(y_t\mid x,f,y_{<t})}{\pi_\theta(y_t\mid x,y_{<t})}, which makes the method a dense-credit replacement for the scalar advantage of GRPO. On LiveCodeBench v6 they report 48.8% for SDPO against 41.2% for tuned GRPO, reaching GRPO's final accuracy with four times fewer generations; even with only correctness feedback the aggregate is 70.2% against 66.6% . Figure 7 shows the self-teaching loop.

Figure 7. Self-teaching in SDPO: after the environment returns feedback on an answer, the same model conditioned on that feedback re-scores the answer, and the token-level log-ratios assign credit (reproduced from ).

RL from text feedback reached a related formulation from a different direction: it optimizes the single-turn return using second-turn, feedback-conditioned rollouts, compiling feedback into first-attempt competence. Its self-distillation variant (RLTF-SD) raises single-turn accuracy on Shortest Path from 0.034 to 0.830, against 0.384 for multi-turn GRPO, and is accompanied by proofs about baseline bias and signal-to-noise in a bandit abstraction . Privileged Information Distillation brought the same idea to multi-turn tool-use agents, where privileged information mined from a stronger model's traces is given to the teacher role of a jointly trained policy; Section 9.1 discusses it with the other agent work.

Later work replaces raw feedback with generated explanations. ROSD has the model write a short reflection contrasting a wrong rollout with the shortest correct sibling, conditions the teacher on that reflection rather than a full solution, and distills only from the first erroneous sentence onward. It reports the best average on SciKnowEval and ToolAlpaca for three backbones, and removing the reflection drops the 4B average from 72.83 back to the SDPO level (67.56). A diagnostic study of critique-conditioned teachers argues that a scoring pass over a reference cannot search for errors, whereas generating a critique with verbatim quotes of the rollout can; its localization and correction scores are near zero or of the wrong sign for reference-conditioned scoring but strongly positive for critique conditioning. CriticOPD and RC-OPD go further and repair the failed rollout: the teacher critiques, a corrected continuation is generated from the last valid point, and only then is the student supervised, with segment-specific hindsight on the error stage. RC-OPD reports mean Avg@4 of 44.17/66.11/66.94 at 1.7B/4B/8B against 40.28/62.50/63.33 for OPSD; it uses an external diagnostic model, which places it at the boundary of strict self-distillation. Multi-rollout OPD gives the teacher the peer rollouts of the same prompt, split into successes and failures, and reports 61.82 mean@8 on LCBv6 for Qwen3-8B against 49.49 for SDPO. ReTeach builds the context from a multi-round reflect-and-retry loop with outcome-aware sample weights.

The same hindsight idea appears with many feedback sources: test execution and rendered visual artifacts in code , unified outcome and hindsight signals for agents , user interactions and corrections , reflection on rare successes and post-failure hints , and proof-assistant feedback in formal mathematics ; a variational treatment casts learning from language feedback as policy distillation .

4.3 Rubrics, skills and memory

For tasks without a reference solution, the third family uses reusable privileged knowledge. RGSD and RuPI put a per-prompt rubric, with positive and negative criteria, into the teacher's context, so the rubric becomes a conditioning string instead of a judge's scalar reward. RGSD reports parity with GRPO on RubricHub medical and science (+6.1 vs +5.9 points; +4.9 vs +4.5) with zero training-time judge calls instead of about one million, one rollout instead of 16, and a lower rise in HealthBench false-claim rate (30.5%→35.1% vs 30.5%→45.1%). RuPI reports HealthBench gains over rubric-reward GRPO at three model scales (for example 0.347 vs 0.268 at 14B) and, under a matched recipe, gains of 0.034–0.079 over using a reference answer as privileged information . These two works establish that soft, natural-language criteria can serve as cc for open-ended generation, where OPSD with reference solutions does not apply.

SKALD replaces the per-instance solution with an answer-filtered skill card (title, principle, usage, common mistakes) shared across problems, adds an annealed cross-entropy term, and gates distillation per problem by a sampled estimate of whether the card helps. It reports avg@8 gains over GRPO of +2.46/+4.85/+12.01 at 0.6B/1.7B/4B, including equal-FLOP comparisons. Procedural Memory Distillation maintains an online three-level memory (experience, insight, behavior) built from the learner's own rollouts and uses it only in the teacher path, so the deployed student is memory-free; it improves SciKnowEval avg@16 over SDPO from 74.4 to 77.2 (Qwen3-8B) and from 69.5 to 73.3 (OLMo3), and a frozen-policy ablation drops to 54.0. On-Policy Context Distillation (OPCD) manufactures cc by extracting experience from unlabeled self-generated traces and distills it on-policy with reverse KL, improving Qwen3-8B math from 75.0 to 79.7 and Sokoban from 9.4 to 53.9; notably, its teacher–student variant beats pure self-distillation (53.9 vs 18.8 on Sokoban), a reminder that a same-model teacher is not always the strongest option.

Other works use the same mechanism to internalize personas , prompt enhancements for image generators , safety constitutions and task-specific soft prompts into weights.

4.4 Label-free privilege

The last family removes external labels from the teacher's context altogether. CANON samples 32 rollouts, takes the majority answer, and places the most confident solution in the majority set into the context of a frozen teacher; every rollout is then distilled against that teacher with a Jensen–Shannon divergence. With no labels, reward model or verifier, it raises AMC avg@32 for Qwen3-4B-Instruct from 66.0 to 76.5, above test-time RL baselines (TTRL 70.3) and close to a gold-conditioned oracle (76.9). u-OPSD makes the same substitution and distills only the rollouts that disagree with the pseudo-solution; it reports non-thinking Qwen3-4B/8B averages rising from 40.96 to 49.49 and from 43.57 to 54.31, above supervised OPSD (46.29/52.04), while 13.3% of its pseudo-labels are wrong. In thinking mode the gain shrinks to a tie with supervised OPSD. CoDA adds a correction channel built from minority trajectories and reaches a level comparable to supervised OPSD on Qwen3-4B (54.57 vs 54.82).

Figure 8. Label-free OPSD: the teacher is conditioned on a majority-vote pseudo-solution from the model's own rollouts, and only disagreeing rollouts are distilled (reproduced from u-OPSD ).

Two further constructions show that the "privilege" need not be information at all. WISD keeps the context fixed and damages the weights (3-bit quantization or magnitude pruning) to obtain a weaker model πw\pi_w; the student is pushed along the log-ratio log⁡π0−log⁡πw\log\pi_0-\log\pi_w, that is, away from what the weakened model would say. LoOPSD obtains a stronger teacher from the same weights by executing a block of layers twice. At the far end, On-Policy Self-Training generates both the questions and the privileged rationales from topic descriptors alone, filters them by re-solvability and an inverse consensus gate, and iterates; on Gemma 4 E2B it lifts MMLU from 37.5 to 47.7 on the same self-generated data on which SFT reaches 40.0.

Label-free OPSD has been extended to test-time policy optimization , self-play and label-free spatial reasoning , and to training on self-generated chains of thought .

Takeaways

  • The content of cc, not the loss, determines what an OPSD method needs and where it applies. References suit verifiable reasoning, feedback suits interactive tasks, rubrics and skills suit open-ended generation, and consensus removes labels.
  • Several independent studies find that less than the full reference (masked, abstracted or averaged views) works as well or better at 4B–8B scale, and one finds that a mismatched reference can outperform the correct one. The teacher's benefit is therefore not simply answer leakage.
  • Label-free contexts recover most of the supervised gain on competition math in non-thinking mode, with gains shrinking for thinking models.

5. Shaping the Signal: Where, How Strongly and in Which Direction to Distill

Once a self-teacher exists, the vanilla objective ∑tD(pt,qt)\sum_t D(p_t,q_t) applies the same divergence, with the same weight, to every token of every rollout of every problem. Most method papers in the corpus change exactly one of these defaults. This chapter groups them by the level at which they intervene: individual tokens or steps (Section 5.1), the divergence and its stability (Section 5.2), and whole problems or rollouts (Section 5.3). Many of these works report gains of one to three points on 30-problem competition benchmarks, often from single runs; Table 5 at the end of the chapter records the evidence each one provides.

mermaid
mindmap root((Signal shaping)) Token and step weights Semantic spans · 37 Position schedules · 92 Adaptive horizons · 38 Sparse token budgets · 93, 94 Per-token teacher choice · 95, 96 Divergence and stability Control variates · 39, 40 Trust regions · 97 Entropy-aware direction · 98 Sign of the update · 21, 99 Supervised anchors · 100 Problem and rollout level Pass-rate weighting · 41, 101 Correctness filtering · 42 Teacher neighborhoods · 102

Figure 9. Subtree of Chapter 5: interventions on the distillation signal grouped by the level at which they act.

5.1 Token and step selection and weighting

A first group of methods argues that the teacher's signal is informative only on a small subset of tokens. STEPS annotates semantic spans of each rollout (key steps and error spans, at most a quarter of the tokens), applies forward KL on key spans and reverse KL on error spans, and decays the distillation term to zero by step 40 while RL continues; on Qwen3-8B it reports a five-benchmark average of 81.51 against 78.75 for GRPO (paired 95% interval [0.41, 4.91]), although with the policy as its own annotator the interval includes zero. Position-weighted OPSD multiplies the per-token loss by a monotone sigmoid of normalized position, motivated by the observation that the teacher's preferences become more reliable as the rollout commits to a path. The authors describe the resulting gains of about one point as modest. DASH replaces the constant 1/T1/T coefficient with a sequence-conditioned weight produced by a gated backward recursion over local divergences. Against a matched four-seed OPSD rerun it reports macro Avg@12 of 45.07/65.00/66.40 at 1.7B/4B/8B (OPSD: 41.87/63.60/64.80), and its ablation attributes most of the gain to where weight is placed rather than to a larger average coefficient.

*Figure 10. Divergence-adaptive supervision horizons in DASH

3838

: local teacher–student divergences on the student's rollout are turned into per-token coefficients by a gated backward recursion (reproduced from the original paper).*

Taken to its limit, selection becomes a token budget. A study of gradient estimation in on-policy distillation shows that tokens selected for gradient-estimation reliability, in addition to usefulness, can carry the signal at a budget of 0.1% of tokens (about one token per trajectory), where the selected subset outperforms full-token OPD on four competition benchmarks. The study uses a separate teacher, but its conclusion transfers directly to the self-teacher setting. BiToK-SD learns a continuous per-token weight through a bilevel threshold and reports small gains over OPSD (41.48 vs 40.85 at 1.7B; 64.85 vs 64.64 at 8B). Two methods construct a different teacher per token. MTA-OPSD builds a pool of teachers with different exposures of the reference and lets each token choose the KL-farthest teacher consistent with a voted supervision direction. AVSD combines a family of privileged views (full solution, partial solution, final answer; or reference code, hint, execution feedback) into a geometric consensus target with a bounded teacher-specific residual, and reports that the share of incorrect-sign credit among the highest-magnitude tokens of incorrect rollouts falls from 12.3% (OPSD) to 2.9%.

Credit-oriented variants use the same token-level quantity to reweight an RL advantage rather than as a distillation target. CSCR down-weights tokens whose likelihood the privileged self-conditioning changes most, renormalizing within each response so the total credit is preserved, and reports gains over GRPO in all ten model–benchmark cells it evaluates. The same token-selection logic recurs in position-selective judge training , Fisher-informed recalibration of feedback-based targets and reliability-weighted self-distillation on self-generated responses .

5.2 Divergence choice and stability

The second group changes the divergence itself, or its estimator. Because OPSD usually computes only the sampled-token log-ratio, the per-token reward log⁡qt(yt)−log⁡pt(yt)\log q_t(y_t)-\log p_t(y_t) has high variance. KL-for-a-KL subtracts a detached, closed-form value baseline (estimated from the top-kk vocabulary) from that reward and reports matching full-vocabulary accuracy (33.0 vs 33.1 average at 1.7B) with up to 57.7% lower wall-clock time and gradient norms one to two orders of magnitude smaller. CREDIT subtracts a batch-contrastive baseline, the teacher's log-probability of the same token under other prompts with the same feedback. The authors prove that the raw reward is a Bayesian filtering increment that sums to the pointwise mutual information between response and feedback, so the baseline removes credit that the feedback would assign regardless of the prompt. They report 67.0 vs 66.0 for SDPO on Qwen3-8B at essentially unchanged step time.

Trust Region OPD partitions tokens with a stochastic gate min⁡(πT/πS,1)\min(\pi_T/\pi_S,1) into trust-region tokens (reverse KL) and outliers (top-kk forward KL), and shows that the forward-KL term alone collapses; the evaluation uses a separate teacher. E²-OPSD diagnoses entropy overshoot in vanilla OPSD and chooses forward or reverse KL per token from the sign of the teacher–student entropy gap, damped by its magnitude; combined with a retrieved exemplar in place of the current reference, its ablation chain goes from 44.64 (OPSD) to 48.89 on Qwen3-1.7B thinking-mode math. Adaptive Supervised Anchoring adds a separate teacher-forced cross-entropy pass on the canonical solution, weighted online by rollout–target similarity, after showing that the teacher's target-token accuracy falls from 80.4% to 3.8% as the student prefix is corrupted. It therefore treats the reliability of qtq_t on off-track prefixes as the central failure.

Two works question whether the update should point toward the teacher at all. Anti-Self-Distillation ascends rather than descends a Jensen–Shannon divergence to the self-teacher, gated by teacher entropy, and interprets the signal through pointwise mutual information; on five models it reports +2.1 to +11.5 average points over GRPO and two- to ten-fold step speedups, while in the same pipeline the default descending self-distillation underperforms GRPO on every model. DCSD decouples credit direction from magnitude: the sign of step credit comes from a belief probe and the magnitude from a non-teacher quantity, with four theorems under stated calibration assumptions. A diagnostic paper supports this skepticism from the measurement side. The implemented two-sided privileged score separates correct from incorrect AIME trajectories only at chance (AUC 0.505), and all five token-score configurations it trains with fall 30–40 points below outcome-only GRPO, while SDPO in its native setting serves as a positive control. Together these results indicate that the privileged log-ratio is a likelihood quantity whose alignment with outcome value must be checked, not assumed; Chapter 8 develops this point.

5.3 Problem- and rollout-level weighting

The third group intervenes on which problems and rollouts are distilled. PACED weights each problem by a pass-rate kernel w(p)=pα(1−p)βw(p)=p^{\alpha}(1-p)^{\beta}, estimated from KK student rollouts, and proves that the Beta kernel has minimax efficiency at least 1−δ21-\delta^2 under its modeling assumptions. In its self-distillation track it reports MATH-500/AIME24/AIME25 of 93.7/31.6/25.1 against 88.9/25.3/16.9 for unweighted reverse KL, with forgetting on MMLU of 0.6% against 2.2%. A pass-rate-weighted SDPO arrives at w=p(1−p)w=\sqrt{p(1-p)} from a cumulant-generating-function argument and reports +3.2 mean@16 over SDPO on Qwen3-8B. The two papers agree that problems the student always solves or never solves carry little useful self-distillation signal, the same zone-of-proximal-development argument that motivates difficulty filtering in RLVR.

C-OPSD removes rollouts before any forward pass: it samples one response per problem, checks its answer symbolically, and distills only a selected subset of the incorrect ones. It matches full OPSD at all three scales (differences of −0.19 to +0.23 points) while distilling 26–29% of responses and shortening wall-clock time by about 40%, which shows that much of vanilla OPSD's compute is spent on rollouts that contribute little. Neighborhood OPSD replaces the single teacher with an offline-selected pool of 25 perturbed teachers and routes one per student-visited state, reporting +1.67 to +2.75 Avg@12 over OPSD across three runs per method.

MethodLevelWhat changesModel / taskReported comparison
STEPS spanKL on ≤25% annotated spans, direction per span type, decayedQwen3-8B, five math benchmarks81.51 vs GRPO 78.75; paired CI excludes 0
DASH tokenbackward-recursive per-token coefficientQwen3 1.7B–8B, AIME/HMMT66.40 vs OPSD 64.80 at 8B; 4 seeds for OPSD
BiToK-SD tokenbilevel continuous token weightsQwen3 1.7B–8B, AIME/HMMT64.85 vs OPSD 64.64 at 8B; no variance
AVSD tokenconsensus over multiple privileged viewsQwen3-8B, math and code62.1 vs OPSD 59.2
KL-for-a-KL estimatorclosed-form value baseline1.7B student, separate teacher33.0 vs 33.1 full-vocabulary, up to 57.7% less wall-clock
CREDIT estimatorbatch-contrastive baselineQwen3-8B, science and code67.0 vs SDPO 66.0
E²-OPSD divergenceentropy-gap-directed forward/reverse KLQwen3-1.7B thinking, math48.89 vs OPSD 44.64
AntiSD directionascend rather than descend divergencefive models, mathQwen3-8B 65.7 vs GRPO 57.4
PACED problempass-rate Beta kernelQwen2.5-Math-7B, MATH/AIMEAIME25 25.1 vs 16.9 unweighted
C-OPSD rolloutdistill only selected incorrect rolloutsQwen3 1.7B–8B, AIME/HMMTparity with OPSD at ~40% less wall-clock

Table 5. Representative signal-shaping methods. "Reported comparison" gives the headline figures the authors report against their closest baseline in their own setting; benchmarks, budgets and checkpoint-selection rules differ across rows, so values are not comparable across rows.

Takeaways

  • The vanilla uniform objective leaves room on three axes: which tokens, which divergence, and which problems. Most reported improvements fall in the one-to-three-point range on small competition benchmarks.
  • Methods that concentrate the signal (on spans, late positions, informative tokens or mid-difficulty problems) agree that most tokens and most problems contribute little. C-OPSD shows that distilling only 26–29% of the responses costs no accuracy.
  • Several works find that the privileged log-ratio is not reliably aligned with outcome value. This motivates variance-reducing baselines, decoupled direction and magnitude, and in one case reversing the update.

6. Teacher Dynamics: Frozen, Moving, Trained and Ahead

Because the teacher and the student share weights, OPSD must decide which weights play the teacher. A teacher frozen at θ0\theta_0 never improves, so the student can at most absorb what the initial model already expresses under privileged context. A teacher that tracks the student improves with it, but it also inherits the student's drift and can reinforce its own errors. The literature has explored the whole range between these poles, and Figure 11 places the main options on one axis.

mermaid
flowchart LR F[Frozen initial teacher] --> E[EMA or trust-region teacher] E --> G[State-gated refresh] G --> T[Teacher trained by its own objective] T --> L[Lookahead or future teacher] F -.-> A[Architectural variant: looped layers]

Figure 11. Teacher-update strategies in OPSD, ordered from least to most coupled to the student's trajectory. Arrows indicate increasing coupling, not chronology.

6.1 Frozen and slowly moving teachers

The two founding recipes made opposite choices: the Self-Distilled Reasoner freezes the teacher at θ0\theta_0 for stability and implicit regularization, while SDFT and SDPO use an EMA of the student, with SDPO also deriving a closed-form trust-region interpolation between the current and the initial teacher. A study of temporal coupling isolates this variable. With a fixed refresh period of 50 steps, the best 100-step score (0.755) is followed by collapse at step 189 of a 300-step run. A state-gated refresh rule (CGTR), which updates the teacher only when an isolation gate, a reward ratchet and a length-tail gate all agree, reaches 0.806 with three refreshes and no collapse in three seeds (fixed refresh: three collapses in three seeds). The paper's broader point is that when the teacher moves is a stability variable, not an implementation detail.

Figure 12. Teacher-update strategies compared in the temporal-coupling study (reproduced from the original paper).

6.2 Teachers trained by their own objective

A second line stops treating the teacher as a by-product of the student and gives it its own learning signal. VISTA adds a separately parameterized teacher that is pulled toward the student's distribution, but only on verified-correct rollouts and only where the student is confident. It reports 66.9 vs 64.8 for OPSD at 8B, with smaller differences at 1.7B and 4B. DualOPSD trains the teacher LoRA after each student step to match the updated student shifted by the initial model's privileged log-probability residual, so the teacher keeps its privileged advantage while following the student. Its reported gains over OPSD are large at 4B and 8B (12.3 and 15.7 points) and persist at measured compute parity, but they are limited to Qwen3 in non-thinking mode on math. PAST trains the self-teacher on complete correct and failed student trajectories in each cycle before freezing it to supervise the student, and reports +5.6 macro Avg@12 over vanilla OPSD at 1.7B with a 95% interval of [2.6, 8.3].

Other works make the teacher an RL learner. TESD gives the teacher its own rollouts and a reward, optimizes it with a GRPO-style objective, and gates the student's distillation per query by the teacher's mean reward. It reports small gains over GRPO (74.32 vs 73.40 on Qwen3-4B), and its naive SDPO+GRPO sum lands between the two components. JOLT optimizes both roles in one shared model, with an outcome reward on teacher rollouts and a KL toward the unprivileged student, and reports MATH 72.59 vs 68.36 for GRPO with 13.0× fewer completion tokens to reach 63.44% accuracy. HATCH has the policy learn to produce the hints it later consumes, anneals hinted advantages as reward contrast appears, and projects out the hint-generation gradient that would conflict with solving.

6.3 Lookahead and future teachers

The third option makes the teacher a future version of the student. B-OPSD runs 50 temporary OPSD updates to obtain πF\pi_F, freezes it, restarts the student from π0\pi_0 and distills from πF\pi_F. With rollout-based privileged information it raises the matched-baseline average from 27.50 to 41.30 on Qwen3-4B and from 48.80 to 64.44 on Qwen3-8B, in single runs. Near-Future Policy Optimization uses a checkpoint a few dozen steps ahead as guide, refreshed by an adaptive controller, and finds a near-future teacher (63.23) better than a far-future one (61.92) after identical continued RL. LEAD builds lookahead teachers that score each instance with what the model would learn from other on-policy rollouts, and keeps out-of-distribution performance near GRPO in settings where SDPO degrades. LoOPSD keeps the weights but changes the computation: it searches for a contiguous block of layers to execute twice, adopts the loop only on strict validation improvement, and reports a looped teacher at 50.87% against 26.38% for the standard path after 1,200 updates on Qwen3-4B.

The breadth of the line includes EMA and variational teachers for language feedback , self-evolving teachers for agents and industrial code models , and label-free spatial reasoning with a moving teacher .

Takeaways

  • A frozen initial teacher is the stable default, but it caps what the student can learn at what the initial model expresses under privileged context.
  • Moving the teacher helps only when the update is controlled: gated refresh, verification-filtered or residual-preserving teacher updates, or a separate teacher objective. Unconditioned periodic refresh collapsed in the one controlled multi-seed study.
  • Future-checkpoint teachers report some of the largest gains in the corpus, mostly from single runs on Qwen3 math. Replication across seeds and model families is the outstanding need.

7. Coupling Self-Distillation with Reinforcement Learning

Pure OPSD has no outcome signal: the student moves toward whatever the privileged teacher prefers, whether or not that preference leads to a correct answer. RLVR has the opposite profile. Its reward is sparse and its advantage vanishes on uniformly solved or uniformly failed groups, but it is anchored to the verifier. Most of the 158 papers the classification places on the RL-coupling axis try to keep the verifier as the source of direction and use the self-teacher for density. The resulting designs differ mainly in where the two signals meet: in the loss, in the advantage, or in a routing decision made before either is computed (Figure 13).

mermaid
flowchart TB R[Rollout group with verifier rewards] --> S{Where do the signals meet} S --> L[Loss level: RL loss plus distillation term] S --> A[Advantage level: teacher ratio rescales or shifts the RL advantage] S --> RT[Routing level: each rollout or prompt gets one signal] L --> L1[HDPO, SDPG] A --> A1[RLSD, RLCSD, CEPO, SIPO] RT --> R1[SRPO, I-SDPO]

Figure 13. Three ways of combining a verifier reward with a self-teacher, with representative methods from Section 7.1–7.3.

7.1 Additive losses

The most direct combination adds a distillation term to the RL loss. HDPO applies it only on "cliff" prompts where all KK standard rollouts fail, conditions the teacher on the ground-truth answer, keeps only teacher rollouts that score 1, and adds a top-kk JSD term to GRPO. The gains are small and trade off against each other: at weight 0.1, pass@8 improves by 1.4–1.7% while pass@1 falls by 2.3–2.8%. Self-Distilled Policy Gradient keeps exact full-vocabulary reverse-KL distillation as an auxiliary loss, gated to positively advantaged rollouts and annealed, and reports that the pure-OPSD baseline collapses at 1.7B (AIME24 0.017 at the last checkpoint) where the combined method does not. Self-Distillation Zero trains a reviser first: a supervised phase on filtered self-revision traces teaches the model to improve its own attempt given a one-bit correctness signal, and that reviser then serves as the privileged teacher. On Qwen3-4B-Instruct the average rises from 49.8 to 60.3, above RFT (54.3) and GRPO (53.1) at a comparable generation budget.

7.2 Teacher ratios inside the advantage

The second design treats the teacher–student ratio as a token-level modulation of an RL advantage whose sign comes from the verifier. Self-Distilled RLVR (RLSD) states the motivation directly. Distribution matching to a privileged teacher leaks answer-specific behavior: leakage counts rise monotonically over 100 steps of OPSD while validation accuracy peaks at 10–20 steps and then declines. RLSD therefore uses the clipped, stop-gradient evidence ratio qt(yt)/pt(yt)q_t(y_t)/p_t(y_t) only as a magnitude multiplier on the verifier advantage, and reports 56.18 average accuracy on multimodal reasoning against 53.86 for GRPO and 52.49 for OPSD.

Figure 14. Self-Distilled RLVR keeps the environment reward as the source of update direction and uses the privileged teacher only to set per-token magnitude (reproduced from ).

Contrastive variants then remove the student from the ratio. RLCSD contrasts the frozen teacher conditioned on a correct hint with the mean of the same teacher conditioned on KK wrong hints, so that answer-independent preferences cancel. It beats GRPO and four OPSD baselines on the averages of both task families for all four backbones it tests (for example Knights-and-Knaves 68.2 vs 63.1 at 4B). CEPO uses the ratio PT+(yt)/PT−(yt)P_T^+(y_t)/P_T^-(y_t) between teachers conditioned on an accepted and a rejected answer, proves that this anchors the update direction and contains RLSD as a special case, and in its short geometry runs finds OPSD and SDPO below the untrained baseline. SIPO adds the clipped difference of two self-teacher evaluations (reference answer versus an in-group mistake) to the group advantage and reports gains over GRPO and RLSD on math, science and code. Outcome-guided OPSD uses forward KL on correct trajectories and reverse KL on truncated incorrect ones, and APSD makes the target a correctness-conditional product of experts that interpolates toward the teacher after success and extrapolates beyond it after failure (LCBv6 49.62 vs 45.04 for SDPO at 8B). β-OPSD derives OPSD from a KL-regularized policy-optimization objective in which vanilla OPSD corresponds to β=1\beta=1, and trains toward a scheduled interpolant on the geometric path between reference and teacher. It reports a 5.74-point gain over vanilla OPSD at 1.7B.

7.3 Routing

The third design decides, per rollout or per prompt, which signal applies. SRPO sends correct rollouts to GRPO and incorrect rollouts with a usable teacher to SDPO, reweighting the distillation branch by teacher confidence. Its ablation shows that advantage-level mixing instead of routing loses 3.3 points, and its five-benchmark average at 8B is 77.4 against 74.0 for GRPO and 71.1 for SDPO, at lower per-step time than either. I-SDPO routes at the prompt level: all-incorrect groups with available ground truth get distillation, and any group with a success gets pure GRPO. It reports 70.31% average mean@16 on SciKnowEval against 56.67 for GRPO and 65.74 for SDPO, with propositions on a bias floor and self-annealing that it states hold only locally. Policy reheating uses a short OPSD stage for a different purpose: to restore entropy after RL has collapsed it, making a smoothed distribution the policy's default instead of raising the sampling temperature.

Figure 15. Sample routing in SRPO: correct rollouts receive the group-relative advantage and failed rollouts with a usable teacher receive self-distillation (reproduced from ).

The RL-coupling axis is the largest in the corpus. Further hybrids weight distillation by verifier confidence or uncertainty , combine test-time RL with self-distillation , target unit-test generation with step-level credit , and unify outcome and hindsight feedback for agents .

MethodCombination levelTeacher contextVerifier roleHeadline comparison reported
HDPO lossground-truth answerselects cliff prompts, filters teacher rolloutspass@8 +1.4–1.7%, pass@1 −2.3–2.8% vs GRPO
SD-Zero loss (after reviser SFT)own attempt + correctness bitgives the bit60.3 vs GRPO 53.1 (Qwen3-4B)
RLSD advantage magnitudereferencesets direction56.18 vs GRPO 53.86
RLCSD advantage magnitudecorrect vs wrong hintssets directionK&K 68.2 vs 63.1 (4B)
SIPO advantage shiftreference vs in-group mistakesets directionmath 59.4 vs GRPO 57.3
APSD target constructionfeedbackchooses interpolate vs extrapolateLCBv6 49.62 vs SDPO 45.04 (8B)
SRPO rollout routingcorrect siblingroutes rollouts77.4 vs GRPO 74.0 (8B)
I-SDPO prompt routingground truthroutes prompts70.31 vs GRPO 56.67

Table 6. Representative OPSD–RL hybrids. Comparisons are each paper's own headline setting and are not comparable across rows.

Takeaways

  • The dominant pattern lets the verifier set the direction of the update and the self-teacher set its token-level magnitude. Several papers motivate this by showing that pure distribution matching leaks answer-specific behavior and degrades after early peaks.
  • Contrastive teachers (correct versus incorrect context) cancel preferences that do not depend on the answer, and they are the most common recent design.
  • Routing rollouts or prompts to exactly one signal is simpler than mixing advantages, and in SRPO's ablation it was also better.

8. Foundational Questions

The method chapters report mostly positive results, because method papers tend to. Alongside them, the corpus contains an unusually large body of controlled analyses: 78 papers carry the analysis-and-theory label, and many of them were written to test the assumptions the method papers share. This chapter organizes that evidence around six questions. For each one we state the proposition, present the evidence on both sides with its conditions, and say where the balance currently lies.

8.1 When does OPSD help at all?

The founding results were obtained on instruct models in non-thinking mode, with short rollouts (about 1,024 tokens) and short training runs. Several groups re-ran the published recipes outside that regime and report losses. Rethinking OPSD for Thinking Models trains five thinking models with the original recipe and evaluates them at a 38,912-token budget. All five lose avg@16; for example, Qwen3-1.7B falls from 0.372 to 0.308 (paired 95% interval [−0.095, −0.037]). On the same student, vanilla OPD with a larger external teacher does not lose, which localizes the problem in the privileged teacher rather than in on-policy distillation itself. A broad diagnostic sweep of more than 100 runs finds that under matched student and teacher reasoning modes OPSD mostly loses: Qwen3-4B-Thinking falls from 75.4 to 65.8 Avg@8 while output length grows from 22.8k to 43.4k tokens, and Qwen3.5-4B collapses entirely. Gains appear mainly when a thinking teacher supervises a non-thinking student. In code, a split that separates unseen problems from unseen test cases finds SDPO losing 0.5–9.9 points on held-out LiveCodeBench problems while GRPO gains, and all 24 OPSD variants it measures have gradients negatively aligned with the held-out math policy gradient.

Figure 16. Privileged-teacher next-token probabilities along one Qwen3-8B thinking trace, used in to show where the self-teacher's preferences diverge from the student's reasoning (reproduced from the original paper).

Two further studies narrow the conditions. Hindsight Compacts but Does Not Repair applies OPSD to correct-only or incorrect-only rollouts of thinking models. Correct-only training shortens outputs by 18–29% at small accuracy cost, incorrect-only training costs 6.6–10.0 points, and no variant raises pass@8; the authors read OPSD on thinking models as compression, not repair. Privileged-but-Biased reproduces SDPO's gains in its original SciKnowEval regime but finds that on harder agentic and general-QA tasks the loss falls while validation does not, with 18 of 20 OPSD-recipe cells non-positive. It traces the failure to a measurable bias of the PI-conditioned teacher, scored by a forward-pass-only diagnostic. The Many Faces of OPD isolates one ingredient of that bias. A standalone Qwen3-14B teacher scores 62.1% on GPQA-Diamond, but only 46.0% when it must continue the student's prefixes, so a teacher that is strong in isolation can be a weak supervisor on off-track states.

The positive side is not empty. A predictive law shows that across six constructions of cc the pre-training accuracy gap between student and self-teacher linearly predicts the post-training gain (R2=0.977R^2=0.977 across four Qwen3 sizes; leave-one-out R2R^2 of 0.949 and 0.996 across context types). A study of PI selection for agents likewise finds offline diagnostics that correlate with post-training pass@1 in all nine student settings. A per-node gradient-alignment analysis shows that distillation gradients align with an ideal success-based gradient mostly on incorrect rollouts, and that the best teacher flips with student capacity. These results imply that OPSD helps when, and roughly in proportion to how much, the privileged context makes the same model better on the student's own prefixes. They also explain why it fails when a long, self-consistent thinking trace leaves the teacher little to add.

8.2 What does privileged information actually add?

If the privileged context worked by revealing answer-relevant knowledge, replacing it with an unrelated solution should destroy the gain. Three studies did this and found the opposite. OP²SD gives the teacher a problem and solution from a different training example. It has the highest point estimate in 8 of 9 model–benchmark groups and, at Qwen3-8B, beats both the base model and OPSD on all three benchmarks (AIME 2024: 55.83 vs 45.00 for OPSD and 28.47 for the base model), and the gain survives when a single fixed example is used. Shuffled OPSD finds that an unrelated reference plus a word-shuffled problem statement beats standard OPSD at all three scales, whereas word-shuffling the reference itself does not. Rethinking Privileged Information reaches the same conclusion through a direction decomposition (Section 4.1). An answer-matched suite with reference-free controls is more cautious. With a thinking teacher supervising direct-response rollouts, a reference-free control reaches +1.80 points in-domain against +3.10 for a clean solution, and the 1.30-point difference does not survive multiple-comparison correction.

mermaid
flowchart LR C[Privileged context c] --> B[Changes teacher behavior: length, mode, style] C --> K[Adds problem-specific knowledge] B --> G[Student gain] K --> G B -. tested by .-> U[Unrelated or shuffled references] K -. tested by .-> M[Mismatched or wrong-answer controls]

Figure 17. Two channels through which privileged context can help, and the controls used to separate them in .

A drift analysis supports the behavioral reading. Varying the content of cc (demonstration, feedback, rephrase) moves the student far more than varying its source: the median per-token reverse-KL ratio is 7.2 versus 1.4, and LoRA updates that share content are more similar (cosine 0.571) than updates that share source (0.255). Taken together, the controlled evidence indicates that, at least on competition math with Qwen3 models, much of OPSD's gain comes from the context changing how the teacher behaves (for example, eliciting a more deliberate mode or a different length profile), and less from transferring problem-specific information. The answer-specific channel is real but smaller than the founding papers implied, and its size depends on the student–teacher mode pairing.

8.3 Collapse, diversity and entropy

A recurring failure is a narrowing of the output distribution. SDSD analysis derives the optimum of on-policy self-distillation with sampled demonstrations. Instead of tilting the base policy by reward, it tilts by expected pointwise conditional mutual information with the demonstration, so equally correct rollouts receive unequal weights. Empirically, the method beats GRPO on Pass@1 but loses on Pass@16, a signature of diversity loss. Thinking collapse identifies a drop in the density of exploratory tokens and repairs it by interpolating toward a frozen base prior inside a high-entropy "sandbox" of tokens (51.8 vs 47.7 for OPSD at 1.7B). A study of clipping shows, with matched clipped/unclipped twins and two proved propositions, that pointwise clipping of the forward KL can reverse the correction and produce 25–450× more terminal repetition loops. A theory of divergence and entropy proves that, at a local optimum of forward KL with a softmax head, student entropy equals teacher entropy plus the residual KL, so the choice of divergence, not the on/off-policy choice, governs whether entropy inflates or deflates. A study that removes the RL loss entirely observes that repelling the student from a solution-conditioned teacher raises accuracy only by re-activating a hybrid thinking mode, and always ends in length blow-up and truncation collapse. The critical review in organizes these failures as one symptom (collapse) with three levers: the geometry of the signal, the nature of the privileged information, and what anchors the student. Its framing matches the evidence above.

8.4 Forgetting and continual learning

SDFT motivated OPSD as a continual-learning method, and a theory paper supports it in an idealized setting. In a one-step softmax model over a clustered chain-of-thought graph, self-distillation with edge-level hints provably retains old-task success after any number of later-task updates, while SFT provably forgets. The empirical picture is more mixed. Denser ≠ Better finds that in sequential post-training over four domains GRPO ends at 69.90 average against 61.60 for the base model, while SDPO ends at 52.55 and a variant collapses. It proves that the teacher-projected token target incurs excess KL relative to the minimum-drift policy at matched success; on-policy sampling, the authors argue, decides which histories are supervised but does not by itself confer RL's conservative bias. Off-policy grafting Pareto-dominates both SFT and OPSD on new-versus-old task axes by learning an update on an earlier checkpoint and applying a masked copy of it. A three-axis decomposition (rollout source, teacher coupling and KL direction) shows that coupling the teacher to the student gives a peak followed by collapse. The resolution suggested by these results is conditional: OPSD retains better than SFT when the target task is close to what the model already does, but it is not automatically less forgetful than RL, and it can drift more at equal success.

8.5 Signal strength and scaling

Why do gains shrink with model size? OASIS measures recovery directly: on failed prefixes the gold-context teacher resumes to a correct answer 77% of the time at 1.7B against 37% for the student alone, but the gap shrinks to 17 points at 4B. Its fix, supervising verified trajectories with another of the model's own rollouts as context, recovers +0.6 to +3.1 points over OPSD as scale grows. A dynamical analysis factorizes the loss-decay rate into a learning signal and an occupancy factor. Self-teachers obtained by RL close 71–77% of the teacher–student gap, larger external teachers only 13–48%, and the authors prove a local convergence bound in which the vanishing learning signal is the limiting factor. Both papers point to the same constraint: a self-teacher helps only as long as there remains a gap between what the model can do with cc and without it, and that gap closes as models and training progress.

8.6 Alignment risk

Self-distillation can also move properties nobody measured. Recursive self-distillation on benign data with a benign cue raises SORRY-Bench compliance from 26% to 94% (Phi-4 14B) and from 31% to 97% (Qwen3 14B) while GSM8K and IFEval are unaffected. SFT on the same safe answers and a frozen teacher do not move compliance, and the effect grows with scale. The setting is off-policy and recursive rather than the usual single-pass OPSD, but it shows that a self-teacher can amplify a context-induced behavior shift over iterations. The finding connects directly to Section 8.2: if the context mainly changes teacher behavior, then whatever behavior the context induces is what gets distilled.

QuestionEvidence that OPSD works as claimedEvidence against or qualifyingCurrent balance
Does it help?founding gains on non-thinking math and science ; predictive law losses for thinking models and held-out problems helps when the self-teacher gap on student prefixes is large; often hurts matched thinking models
What does cc add?correct references beat wrong-answer controls in some settings unrelated or shuffled references match or beat correct ones largely behavioral at 4B–8B math; knowledge share is smaller and setting-dependent
Diversitydense signal with low sampling cost Pass@k loss and thinking collapse ; clipping loops collapse is common and needs explicit anchoring
ForgettingSDFT retention ; graph-model theory more drift than GRPO in sequential training ; grafting dominates better than SFT, not reliably better than RL

Table 7. Summary of the evidence on four foundational questions. "Current balance" is this survey's reading of the cited evidence, limited mostly to Qwen-family models up to 14B.

Takeaways

  • OPSD's gains are conditional. They are strongest when a privileged context makes the same model clearly better on the student's own prefixes (non-thinking students, short rollouts), and they often reverse for thinking models evaluated at long budgets.
  • Controlled studies show that unrelated or shuffled privileged contexts can match the correct reference. Much of the teacher's value is behavioral rather than answer-specific.
  • Diversity collapse, clipping-induced loops and drift in continual training are recurring failure modes, and anchoring to a base prior or to an outcome signal is the common remedy.

9. Applications

OPSD spread beyond competition math within months. Each new domain had to answer the question from Chapter 4 in its own terms: what can the training-time teacher see that the deployed model cannot? Agents answer with hindsight about the trajectory, vision models with a better view of the input, diffusion models with a later state of their own denoising, safety training with a safety-oriented context. This chapter surveys those adaptations and, for each domain, what changed relative to the reasoning recipe.

9.1 Agents and multi-turn tool use

Multi-turn agents are the largest application domain in the corpus (107 papers), because environment interaction produces abundant privileged information after the fact: which tool call failed, which observation mattered, which turn derailed the episode. Privileged Information Distillation showed early that the picture is model-dependent. On τ-Bench and Travel Planner its joint teacher–student training reaches 44.1% against 32.3% for SFT-plus-RL on Travel Planner with Qwen3-8B, while OPSD is best only on the out-of-distribution airline split and results are mixed on other backbones. Skill-SD summarizes completed trajectories into natural-language skills that condition only the teacher, re-synchronizes the teacher from the student, and reports 64.9% on AppWorld and 62.5% on Sokoban against 50.9% and 51.6% for GRPO; its ablation shows a frozen teacher plateauing 12–16 points lower.

Figure 18. Skill-conditioned self-distillation for multi-turn agents: skills summarized from the student's own trajectories enter only the teacher prompt (reproduced from Skill-SD ).

The main agent recipes after Skill-SD differ in where the privileged context comes from and which steps receive it. OPID extracts a workflow-level rule and step-level advice at up to five critical timesteps from each completed trajectory. SEED has the current policy write the skill itself and regenerates it every update, reporting ALFWorld 91.8% at 3B against 75.0% for GRPO. SCOPE distills only critical error steps with a bounded log-ratio and reports 97.7% ALFWorld success at 3B and 94.0% unseen at 32B. HINT-SD and TurnSight restrict hindsight to failure-relevant turns or aggregate it per tool turn. HINT-SD raises BFCL v3 Avg@4 from 31.56 (best baseline) to 41.88 at less than half the per-step time of a full-trajectory alternative. SD-Search applies a hindsight JSD only at search-query tokens on top of GRPO and reports 0.428±0.008 average EM over five seeds against 0.404±0.008 for AutoRefine. SDAR keeps GRPO unchanged and adds OPSD as a sigmoid-gated auxiliary loss. Without that gate, it reports, naive OPSD collapses on Search-QA and GRPO+OPSD collapses at 1.7B.

These collapses are not isolated. Self-Practice finds that pure OPSD falls below the untrained base on AppWorld and SWE-bench in all four of its settings, and moves the privileged information from the loss to the sampler: it decides only under which context a group of rollouts is drawn, and all groups are trained with GRPO. A study of retrieval-interleaved search agents finds feedback-augmented self-distillation never sustains gains, even though the feedback-conditioned teacher is stronger at inference. With EMA regularization it reaches 0.206 average EM against 0.348 for GRPO. SELF uses the realized environment feedback twice, as teacher context and as a prediction target, and reports gains over both GRPO and SDPO on τ-bench and AppWorld. The common thread is that agents gain most when distillation is targeted at a few steps and gated by an outcome signal; whole-trajectory OPSD is often harmful in long-horizon settings.

MethodPrivileged contextSteps distilledCoupling with RLRepresentative result
Skill-SD skills from student trajectoriesall action tokensimportance-weighted reverse KL plus GRPOAppWorld 64.9 vs GRPO 50.9
SDAR hindsight contextgated per tokenauxiliary loss on unchanged GRPOWebShop 82.8 vs 72.6 (7B)
SD-Search hindsight on querysearch-query tokens onlyauxiliary JSD on GRPOEM 0.428 vs 0.404 (5 seeds)
HINT-SD analyzer feedbackfailure-relevant turnsauxiliaryBFCL v3 41.88 vs 31.56
OPID extracted workflow + step advice≤5 critical stepsauxiliaryALFWorld 84.3 vs 75.0 (3B)
SEED self-written skill, refreshedtrajectoryauxiliaryALFWorld 91.8 vs 75.0 (3B)
SCOPE step-specific skillcritical error stepsbounded auxiliaryALFWorld unseen 89.6 vs 66.4
Self-Practice analyzer guidancenone (sampler only)GRPO onlyAppWorld 16.7 vs 10.6 (4B)

Table 8. Representative OPSD methods for agents. Results are each paper's headline comparison against its own GRPO baseline and are not comparable across rows.

GUI and computer-use agents form a sub-line. GUI-SD conditions the teacher on step-level reasoning and memory privileges and reports AndroidWorld Pass@1 of 67.2 against 52.6 for GRPO. ComputerSD regenerates guidance after each executed step from pre- and post-action screenshots and gates distillation by an analyzer value score (OSWorld-Verified 39.8 vs 37.9). GUI grounding turns the ground-truth coordinate into a visual privilege, a drawn bounding box with a soft zoom, and reports ScreenSpot-Pro 60.7% at about a quarter of GRPO's training time. Further agent work covers persistent consistency regularization , comparative outcome-verified self-distillation , hindsight from environment observations , reminder distillation for coding agents , granularity-adaptive advantages , hierarchical supervision allocation for long horizons , self-retiring distillation , closing the multi-turn conversation gap , proactive dialogue and stopping behavior for GUI agents .

9.2 Code and software engineering

Code provides rich, structured feedback, which is why SDPO's strongest results were on LiveCodeBench . For repository-level software engineering, SWE-Ladder shows the teacher only the slice of the verified repair that matches the current stage (changed files for localization, fail-to-pass tests for reproduction, the diff for patching) and introduces the views progressively. On R2E-Gym it raises pass@1 from 9.6% to 16.4%, against 12.4% for full-patch GRPO+OPSD, in single runs. Hindsight Hint Distillation places a one-sentence hindsight hint at the start of an otherwise self-generated trajectory and reports SWE-bench Verified 51.2% for a 72B model against 44.0% for naive rejection fine-tuning. Other code work includes trajectory intervention , think-mode advantages , difficulty-gated teacher guidance , self-extrapolating policy distillation and memory-augmented text-to-SQL .

9.3 Multimodal understanding

In vision–language models the privileged channel is usually perceptual: the teacher sees the input better. Vision-OPD lets the model's crop-conditioned distribution supervise its full-image distribution, internalizing zooming into a single pass. It reports an average of 79.68 for a 9B model against 73.37 for its base and 77.44 for a 397B model of the same family, and it beats SFT, GRPO, DAPO and OPSD at both scales tested. RP-OPSD obtains the privilege by giving the student a half-resolution image and the teacher the original, which requires only image–question pairs. S²VOPD corrupts the student's view instead of enriching the teacher's, needing no annotations, and lifts a 4B model from 70.68% to 77.44% on six perception benchmarks.

Figure 19. Vision-OPD: the same model conditioned on a zoomed crop serves as teacher for its full-image predictions (reproduced from ).

The same design reappears for time and other modalities. Video-OPSD shows the teacher only the annotated evidence frames and beats standard OPSD in all 15 backbone–benchmark cells, at 60% less training time than GRPO. Clue-OPSD privileges a short clue interval of a long video. ViCuR replaces answer hints with a text description of visual evidence that is recoverable from the image itself, adding about 1.2 points over answer-based OPSD. K-OPSD adapts the approach to engineering drawings with a process-level verifier. The breadth of the line spans streaming and proactive video , distractor-aware temporal privilege , 3D and spatial privilege , audio–visual and speech reasoning , test-time adaptation and visually attributable self-distillation .

9.4 Generative models

OPSD has also been carried to generators that are not autoregressive language models. For diffusion language models, dOPSD uses a later, more-decoded state of the student's own denoising trajectory as privileged information. It is the only method in its comparison that improves over the base model on every benchmark for both backbones (for example, MATH500 from 38.97 to 42.20 on Dream). For image diffusion, D-OPSD continues tuning step-distilled models by placing the target image only in the teacher's condition, improving FID from 48.69 to 40.49 where SFT degrades it to 82.20. DiffusionOPSD turns image-level rewards into bounded targets for intermediate clean-output predictions and reports the best held-out score in 19 of 20 reward-matched settings with 40–63% fewer GPU-hours than DiffusionNFT. OPSD-V uses a cleaner temporal cache as the privilege for few-step autoregressive video generators.

Figure 20. On-policy self-distillation for diffusion language models: the teacher sees a more-decoded state of the same denoising trajectory, and the student learns from its noisier state (reproduced from dOPSD ).

Further generative work covers flow-matching models without a teacher , self-future distillation for diffusion LMs , guided denoiser self-distillation , multi-turn image editing , viewpoint-change video and diffusion planners for robots .

9.5 Continual learning and knowledge injection

Continual learning was the motivation of SDFT , and Section 8.4 discussed the conditions under which OPSD retains old skills. Method work in this area constrains how far the teacher pulls. iSDFT selects, at each prefix, the distribution closest to the student that achieves a budgeted information gain toward the teacher. It improves task skill over SDFT in seven of eight cells while retaining more benchmarks within half a point. MixSD trains on a token-level mixture of expert-conditioned and naive base-model rollouts with plain NLL for knowledge injection, keeping held-out accuracy at the base level (72.4 vs 72.2) where SFT drops to 46.1. CODE teaches causal edits through an open-book self-teacher and avoids the general-capability loss of static editing baselines. Related work covers experiential and self-evolving agents , multi-teacher replay , self-specialized domain teachers , skill-memory internalization and knowledge injection under information asymmetry .

9.6 Alignment, safety and personalization

For safety, the privileged context is a description of how the model should behave, and OPSD distills it while touching as few tokens as possible. OPSA conditions a frozen self-teacher on type-specific safety context and adds a training-free teacher-flip-rate criterion for choosing that context. It reports a 4.0-point composite safety gain and a 3.0-point reasoning gain over matched-data ThinkSafe, that is, a smaller "safety tax".

Figure 21. Safety alignment by on-policy self-distillation: a frozen self-teacher given type-specific safety context supervises the student's own responses (reproduced from OPSA ).

Multilingual Safety via Self-Distillation uses a parallel English query as privilege and cuts the 10-language attack success rate on LLaMA-3-8B from 21.21% to 2.03% with unchanged MMMLU. ERASE defends against multimodal prompt injection by giving the teacher an injection-erased view of the same input. For personalization, GRASP uses per-user rubric aspects as privilege and beats five baselines on LaMP-QA for four backbones. The broader line includes judge training , rubric-based RL , calibration , adversarial self-distillation , policy-based safety alignment and learning from user interactions . Section 8.6 discussed the corresponding risk: recursive self-distillation can also erode safety.

9.7 Efficient reasoning

When the privileged context is an instruction rather than information, OPSD becomes a compression method. CRISP conditions the teacher on a conciseness instruction and periodically refreshes it from the student. Qwen3-14B MATH-500 accuracy rises from 93.0% to 96.3% with 56.3% fewer reasoning tokens, and on AIME 2024 accuracy drops slightly (75.0 to 73.8) at 37.5% fewer tokens. EGRSD adds a teacher-entropy confidence gate to RLSD-style updates and improves the accuracy–length trade-off on six benchmarks. Correct-only OPSD on thinking models acts similarly as a compressor . Further work internalizes chain-of-thought or visual manipulation , prunes long-form reasoning , reduces visual tokens , and trains multi-token prediction and speculative drafters by self-distillation.

9.8 Other domains

The recipe has reached domains well outside language reasoning. Multi-dialect ASR replaces teacher-forced cross-entropy with OPSD toward a gold-transcript-conditioned self-teacher, improving dialect error rates while reducing Mandarin error (3.40% to 3.27%), whereas continued SFT raises it to 4.43%. In generative recommendation, SA-OPSD queries the self-teacher at parent states of the student's own beam search and cuts training cost from 17.77 to 5.79 GPU-hours against a dense alternative. SOLID combines test-time RL with selective OPSD for operations-research modeling, with mixed results across benchmarks. Applications also include vision–language–action robots , navigation , driving policies , retrieval and embeddings , medical agents , financial reasoning and research-plan generation .

Takeaways

  • Each domain re-defines the privilege: trajectory hindsight for agents, a better view of the input for vision and video, a later denoising state for diffusion, a behavioral specification for safety, and an instruction for efficiency.
  • In long-horizon agents, whole-trajectory OPSD is frequently harmful. Successful recipes target a few critical steps and keep an outcome-based RL objective in charge.
  • Perceptual privilege (crops, resolution, evidence frames) is the most consistently positive multimodal design, and it needs no answer labels.

10. Resources: Benchmarks, Frameworks and Systems

OPSD papers share a small set of evaluation settings, and much of the apparent disagreement in the field reflects differences between those settings rather than between methods. This chapter collects the settings and the training infrastructure that recurs in the inspected papers.

10.1 Evaluation settings

Table 9 lists the benchmarks that appear repeatedly among the deeply read papers, with the regime each one represents. Two features of this list matter for interpreting results. First, the most common reasoning setting, AIME 2024/2025 and HMMT 2025, contains 30 problems per benchmark. Differences of one or two Avg@12 points are therefore within the range that seed variation can produce, and several papers report single runs . Second, the founding science/tool settings (SciKnowEval, ToolAlpaca, LiveCodeBench v6) favor SDPO-style rich feedback, while long-budget thinking-model evaluation is where several analyses find OPSD losing (Chapter 8). Comparisons that cross these regimes need care.

SettingRegimeTypical metricUsed by (examples)
AIME 2024/2025, HMMT 2025 (30 problems each)competition math, short rollouts, non-thinking studentsAvg@12 or mean@16
MATH-500, AMC 2023, GPQA-Diamondmath and science, label-free and test-time settingsavg@k, pass@k
SciKnowEval L3 (four domains)science QA with rich feedbackmean@16
ToolAlpacasingle-turn tool useaccuracy
LiveCodeBench v6code with test feedbackaccuracy, pass@k
τ-Bench, Travel Planner, BFCL v3multi-turn tool usesuccess rate, Avg@4
ALFWorld, WebShop, Search-QAembodied, web and search agentssuccess, score, EM
AppWorld, SWE-bench, R2E-Gymlong-horizon software agentstask goal completion, pass@1
OSWorld-Verified, AndroidWorld, ScreenSpot-Procomputer use and GUI groundingPass@1, accuracy
HealthBench, RubricHub, ResearchQAopen-ended generation graded by rubricsrubric score
V*, video QA suites (MLVU, Video-MME)fine-grained and temporal perceptionaccuracy

Table 9. Recurring evaluation settings among the deeply read papers. Inclusion rule: settings used as a headline benchmark by at least two inspected papers; the "used by" column lists examples, not an exhaustive census.

10.2 Frameworks and training systems

Most OPSD implementations in the corpus build on verl with vLLM rollouts. Released code is the exception rather than the rule: among the deeply read method papers, the Self-Distilled Reasoner and ROSD point to public repositories, and many cards record no code, data or checkpoint release. EasyOPD addresses the resulting fragmentation. It separates a user configuration layer, a method-local supervision module and the verl execution layer, connected by five dispatch hooks, so that a method is selected by name and YAML. Its own comparisons also illustrate the regime dependence of Chapter 8: SDPO beats GRPO on SciKnowEval Chemistry (70.4 vs 51.2 after one hour) but loses on GSM8K (87.7 vs 91.8).

Figure 22. The three-layer architecture of EasyOPD, which hosts several on-policy and self-distillation methods behind a common configuration interface (reproduced from ).

OPSD's cost profile differs from RLVR's. It needs one or a few rollouts per prompt instead of a group, but it adds a teacher forward pass over a longer, privileged context, and full-vocabulary variants must also move logits. Asynchronous training is the main systems response. AsyncOPSD recomputes teacher supervision inside the learner on buffered rollouts, bounds staleness with an age cutoff, and reports 1.96–2.25× accepted-sample throughput with slightly higher accuracy than synchronous OPSD, together with bounds on the asynchronous–synchronous objective gap. AsyncOPD , studying separate-teacher OPD, finds that reverse-KL objectives degrade faster with staleness than forward-KL ones, and that an importance-corrected, learner-recomputed surrogate is the most robust. Efficiency also comes from distilling less: C-OPSD distills about a quarter of the responses at equal accuracy , and token budgets of 0.1% can suffice . Related systems work studies fresh rollouts for long horizons , branching rollouts , least-squares policy distillation , budget-aware selective distillation and efficient self-distilled safety alignment .

Takeaways

  • Competition-math results rest on 30-problem benchmarks, and many papers report single runs. Gaps of one to two points should be read as provisional.
  • Results depend strongly on the evaluation regime (rich-feedback science and tools versus long-budget thinking), which accounts for much of the apparent disagreement in the literature.
  • Shared infrastructure is still emerging. Asynchronous pipelines roughly double throughput, and selective distillation removes most of the teacher's cost.

11. Future Directions

The preceding chapters point to open problems that are concrete enough to state as research questions. Each is listed with the early works that point toward it.

Measuring privileged information before training. Whether a context helps is currently discovered by training. Two works show that cheap offline quantities predict the outcome: the pre-training student–teacher accuracy gap and error-localization diagnostics for agent contexts . A forward-pass-only bias score flags harmful contexts , and a teacher-flip rate selects safety contexts . A shared, validated protocol for scoring candidate contexts on held-out prompts would make the design space of Chapter 4 searchable instead of anecdotal.

Separating behavioral from informational privilege. The controls of Section 8.2 show that unrelated or shuffled references can work as well as correct ones . If the teacher's value is largely behavioral, the context can be designed directly as a behavior specification, as in conciseness instructions or skill cards , and answer leakage can be avoided by construction. Methods that explicitly factor the two channels, for example by contrasting matched contexts , are a natural next step.

OPSD for thinking models. The clearest negative result in the field is that OPSD often hurts thinking models evaluated at long budgets . Candidate remedies include anchoring to a base prior in high-entropy regions , supervising only verified trajectories and mismatched-mode teachers. None has yet been shown to turn OPSD into a reliable improvement for long-budget thinking models.

Collapse-resistant objectives. Diversity loss , clipping-induced loops and entropy shifts governed by the divergence are now understood in isolation. What is missing is an objective with guaranteed coverage, for example one whose optimum preserves the relative weights of equally correct responses as RL does, and that is evaluated with Pass@k as well as Pass@1.

Teacher dynamics as a control problem. Gated refresh , trained teachers and lookahead teachers each improve on a frozen teacher in their settings, but they have not been compared under matched compute or across model families. A teacher-update policy that adapts to measured teacher–student gap is an open design.

Scaling and replication. Almost all inspected evidence comes from models of 14B parameters or fewer, often Qwen3, with single runs on 30-problem benchmarks (Chapter 10). Larger models, other families and multi-seed reporting are needed before the one-to-three-point improvements of Chapters 5–7 can be ranked.

Long-horizon agents. Agent results favor targeted, gated distillation over whole-trajectory OPSD , and feedback-augmented self-distillation can fail outright for search agents . A principled account of which steps deserve a privileged target, and how to verify the analyzer that selects them, would unify the many agent-specific recipes.

Behavioral side effects and safety. Recursive self-distillation can amplify a benign context into large shifts in harmful compliance . As OPSD is used for safety and personalization , routine audits of properties outside the training target, such as refusal behavior and calibration , should become part of evaluation.

mermaid
flowchart LR M[Measure privilege offline] --> D[Design context as behavior or information] D --> O[Collapse-resistant objective] O --> T[Adaptive teacher dynamics] T --> S[Scale and replicate] S --> A[Audit side effects] A -. feedback .-> M

Figure 23. A research loop connecting the open problems: offline measurement informs context design, which constrains the objective and teacher dynamics. Results must then be replicated at scale and audited for side effects.

12. Conclusion

On-policy self-distillation turns a language model into its own teacher by giving one copy information the other will not have. Within ten months the idea has produced a large literature that is coherent at the level of its pipeline. Every method chooses a privileged context, a way of shaping the token-level signal, a teacher-update rule and a relationship to the verifier. The choice of context decides where the method applies: reference solutions for verifiable reasoning, feedback and hindsight for interactive tasks, rubrics and skills for open-ended generation, perceptual views for multimodal models, later denoising states for diffusion models, and consensus when no labels exist.

The evidence also supports more specific conclusions than the founding papers could offer. OPSD is most reliable when the privileged context makes the same model clearly better on the student's own prefixes. That holds most clearly for non-thinking students, short rollouts and rich environment feedback, and it often fails for thinking models evaluated at long budgets. A substantial share of the teacher's benefit appears to be behavioral rather than answer-specific. Uniform distribution matching tends to narrow the output distribution and can drift in continual training. The designs that have spread most widely respond to these findings by letting a verifier or outcome signal set the direction of the update and using the self-teacher for token-level magnitude or for a few targeted steps. What remains open is a way to measure the value of a privileged context before training, objectives that are provably resistant to collapse, and evidence at scales and in model families beyond those studied so far.

Appendix A. Scope, Retrieval and Screening

Cutoff and sources. The corpus covers arXiv submissions up to 3 November 2026; retrieval was run on 4 November 2026 over an arXiv-based paper library. Two routes were combined: a keyword route requiring self-distillation or on-policy self-distillation terms together with on-policy training terms in the title or abstract (366 papers), and a semantic route returning the 700 papers nearest to the topic description and submitted after 1 June 2025. Their union contains 822 papers. The tail of the semantic ranking consisted mostly of separate-teacher on-policy distillation and generic knowledge distillation, which was the stopping criterion.

Screening. Each candidate was judged against an inclusion criterion: the teacher distribution must come from the student's own weights under information or inputs the student does not see, and supervision must be applied on the student's own samples. The criterion was calibrated on a sample and then applied to all 822 candidates by an LLM-based judge. The judge included 475 papers, and three known on-topic papers missed by retrieval were added manually, giving a corpus of 478. A small number of included papers use a separate teacher for part of their method; they are labeled as boundary cases and cited only where their findings bear on self-distillation.

Classification and reading depth. Papers were classified on the two axes of Chapter 3 by an LLM-based multi-label classifier. A total of 144 papers were read in full and summarized in structured cards covering mechanism, setting, headline evidence, limitations and figures. They were chosen as the founding works, the most representative method of each leaf, the controlled analyses and the main application papers; mechanism and result descriptions in the text are limited to these papers. The remaining papers are cited from their abstracts, in grouped sentences that state only what each paper says it does.

Limitations. The corpus is limited to arXiv preprints, most of which had not been peer reviewed at the cutoff; some results may change in later versions. LLM-assisted screening and classification can mislabel individual papers. Reported numbers are the authors' own and come from heterogeneous settings, so they are not comparable across papers unless the text says so.

References

[1] Siyan Zhao, Zhihui Xie, Mengchen Liu, et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. 2026. arXiv preprint. arXiv:2601.18734

[2] Jonas Hübotter, Frederike Lübeck, Lejs Behric, et al. Reinforcement Learning via Self-Distillation. 2026. arXiv preprint. arXiv:2601.20802

[3] Idan Shenfeld, Mehul Damani, Jonas Hübotter and Pulkit Agrawal. Self-Distillation Enables Continual Learning. 2026. arXiv preprint. arXiv:2601.19897

[4] Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar, Junpei Komiyama. Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.09228

[5] Qingyuan Zheng, Zhihao Dou. Shuffled OPSD: When Semantic Consistency Hurts On-Policy Self-Distillation?. 2026. arXiv preprint. arXiv:2611.04958

[6] Simran Kaur and Narutatsu Ri and Yinghui He and Liam Fowl and Sanjeev Arora. Rethinking On-Policy Self-Distillation for Thinking Models. 2026. arXiv preprint. arXiv:2607.05184

[7] Yang Li, Gongle Xue, Yuheng Yuan, et al. Diagnosing On-Policy Self-Distillation for Reasoning Language Models. 2026. arXiv preprint. arXiv:2609.39118

[8] Meng Wang, Haohan Zhao, Wenzhuo Liu, et al. Denser ≠\neq Better: Limits of On-Policy Self-Distillation for Continual Post-Training. 2026. arXiv preprint. arXiv:2607.01763

[9] Mingyang Song, Mao Zheng. A Survey of On-Policy Distillation for Large Language Models. 2026. arXiv preprint. arXiv:2604.00626

[10] Fangming Cui, Sunan Li, Jiahong Li. A Brief Overview: On-Policy Self-Distillation In Large Language Models. 2026. arXiv preprint. arXiv:2605.18141

[11] Bowen Zhang. A Formula-Driven Survey and Research Agenda for On-Policy Distillation. 2026. arXiv preprint. arXiv:2606.22793

[12] Justin Robert, Raheel Qader. One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.25936

[13] Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, Furu Wei. On-Policy Context Distillation for Language Models. 2026. arXiv preprint. arXiv:2602.12275

[14] Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, Massimo Caccia. Privileged Information Distillation for Language Models. 2026. arXiv preprint. arXiv:2602.04942

[15] Yuda Song, Lili Chen, Fahim Tajwar, et al. Expanding the Capabilities of Reinforcement Learning via Text Feedback. 2026. arXiv preprint. arXiv:2602.02482

[16] Ken Ding. HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation. 2026. arXiv preprint. arXiv:2603.23871

[17] Chenxu Yang, Chuanyu Qin, Qingyi Si, et al. Self-Distilled RLVR. 2026. arXiv preprint. arXiv:2604.03128

[18] Yinghui He, Simran Kaur, Adithya Bhaskar, et al. Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision. 2026. arXiv preprint. arXiv:2604.12002

[19] Hao Wang, Guozhi Wang, Han Xiao, et al. Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents. 2026. arXiv preprint. arXiv:2604.10674

[20] Qianhao Yuan, Jie Lou, Xing Yu, et al. Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2605.18740

[21] Guobin Shen, Xiang Cheng, Chenxiao Zhao, et al. Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information. 2026. arXiv preprint. arXiv:2605.11609

[22] Dengyang Jiang, Xin Jin, Dongyang Liu, et al. D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models. 2026. arXiv preprint. arXiv:2605.05204

[23] John Gkountouras, Josip Juki'c, Ivan Titov. Consensus as Privileged Context for Label-Free Self-Distillation. 2026. arXiv preprint. arXiv:2607.13643

[24] Phuong Tuan Dat, Qi Li, Xinchao Wang. dOPSD: On-Policy Self-Distillation for Diffusion Language Models. 2026. arXiv preprint. arXiv:2607.04428

[25] Xidong Wu, Qi Qi, Yuqiong Ji, Shangqian Gao. Asynchronous On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.25880

[26] Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, Sean Hendryx. Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models. 2025. arXiv preprint. arXiv:2506.13923

[27] Zizhuo Zhang, Jianing Zhu, Xinmu Ge, et al. Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models. 2025. arXiv preprint. arXiv:2508.00410

[28] Hyeonji Lee, Daejin Jo, Seohwan Yun, Sungwoong Kim. SGPO: Self-Generated Preference Optimization based on Self-Improver. 2025. ACM Trans. Intell. Syst. Technol. 2026. arXiv:2507.20181

[29] Shrenik Bhansali, Larry Heck. Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding. 2025. arXiv preprint. arXiv:2510.05421

[30] Kanghui Tian, Siyuan Liu, Tianxiang Jiang, et al. What Should a Self-Teacher See? Privileged Context Design for On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2609.25623

[31] Ziqi Zhao, Xinyu Ma, Liu Yang, et al. Beyond Imitation: Reflective On-Policy Self-Distillation for LLM Reasoning. 2026. arXiv preprint. arXiv:2605.28014

[32] Deepika Bablani, Ajay Gupta, Wanming Chen. Rubrics as Privileged Information for Open-Ended Generation. 2026. arXiv preprint. arXiv:2608.02948

[33] Yubo Jiang, Fengying Xie, Zhiguo Jiang, Haopeng Zhang. Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.09826

[34] Ye Liu, Srijan Bansal, Bo Pang, et al. Procedural Memory Distillation: Online Reflection for Self-Improving Language Models. 2026. arXiv preprint. arXiv:2607.01480

[35] Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, Nuno Vasconcelos. On-Policy Self-Distillation without Any Supervision. 2026. arXiv preprint. arXiv:2608.06296

[36] Yecheng Wu, Song Han, Han Cai. Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models. 2026. arXiv preprint. arXiv:2610.38681

[37] Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Lan-Zhe Guo. STEPS: Selective On-Policy Self-Distillation for Reasoning. 2026. arXiv preprint. arXiv:2605.10194

[38] ZhiYan Hou, Xinyu Tang, Hongyan An, et al. DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models. 2026. arXiv preprint. arXiv:2608.06243

[39] Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, Yohan Jo. KL for a KL: On-Policy Distillation with Control Variate Baseline. 2026. arXiv preprint. arXiv:2605.07865

[40] Guobin Shen, Lei Huang, Xiang Cheng, et al. From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation. 2026. arXiv preprint. arXiv:2605.11613

[41] Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He and Zhipeng Wang. PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence. 2026. arXiv preprint. arXiv:2603.11178

[42] Harrison Zhao, Lei Huang, Zhiyuan Zhao, Fei Liu, Wen Shi, Zijian Jin. Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.25842

[43] Haowei Guo, Baolong Bi, Ruicheng Zhang, Cong Li, Bingqian Sun, Zhonghai Wu. When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation. 2026. arXiv preprint. arXiv:2606.03532

[44] Yutong Chen, Guangfu Guo, Zhichao Xu, Kunpeng Liu. DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.26019

[45] Randy Ardywibowo, Arnav Dalal, Jiantao Jiao. A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching. 2026. arXiv preprint. arXiv:2610.10447

[46] Zheng Zhang, Xinyue Tan, Lufei Li, Xinyi Zhang, Yexin Li, Kan Ren. Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models. 2026. arXiv preprint. arXiv:2609.37132

[47] Chuanyu Qin, Chenxu Yang, Qingyi Si, et al. Learning from the Near Future: Temporal Self-Distillation for RLVR. 2026. arXiv preprint. arXiv:2604.20733

[48] Leyi Pan, Shuchang Tao, Yunpeng Zhai, et al. RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2606.11709

[49] Gengsheng Li, Tianyu Yang, Junfeng Fang, et al. Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing. 2026. arXiv preprint. arXiv:2604.02288

[50] Yubo Zhang and Xinhong Ma and Zezhong Tan and Ziqiang Dong. I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization. 2026. arXiv preprint. arXiv:2608.12957

[51] Tommy He, Jerome Sieber, Matteo Saponati. A Predictive Law for On-Policy Self-Distillation From World Feedback. 2026. arXiv preprint. arXiv:2605.30070

[52] Andrei Liviu Nicolicioiu and Mohammad Pezeshki and Aaron Courville. On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity. 2026. arXiv preprint. arXiv:2606.26091

[53] Jinyang Wu and Shuo Yang and Zhengxi Lu and Fan Zhang and Yuhao Shen and Lang Feng and Haoran Luo and Zheng Lian and Shuai Zhang and Zhengqi Wen and Jianhua Tao. SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning. 2026. arXiv preprint. arXiv:2607.14777

[54] Ziyue Wang, Shiqi Huang, Weiwen Xu, Bihan Wen, Xudong Jiang. Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models. 2026. arXiv preprint. arXiv:2608.27065

[55] Yu Fu, Longxuan Yu, Haz Sameen Shahgir, et al. Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2605.15239

[56] Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, Jiachen Sun. CRISP: Compressed Reasoning via Iterative Self-Policy Distillation. 2026. arXiv preprint. arXiv:2603.05433

[57] Zihao Han, Tiangang Zhang, Huaibin Wang, Yilun Sun. Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning. 2026. arXiv preprint. arXiv:2605.11458

[58] Khouloud Saadi and Di Wang. One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.30880

[59] Xuyang Zhao, Liting Zhang, Zichen Xu, et al. Is More Privileged Information Better? From Solution Traces to Problem-Solving Structure in Self-Distilled Reasoning. 2026. arXiv preprint. arXiv:2608.01589

[60] Zhanming Shen, Jintao Tong, Shaotian Yan, et al. Purified OPSD: On-Policy Self-Distillation Without Losing How to Think. 2026. arXiv preprint. arXiv:2607.02234

[61] Kaixiang Zhao, Tianrun Yu, Shangzhe Li, et al. Recovering Reliable Supervision via Critique-Guided On-Policy Distillation. 2026. arXiv preprint. arXiv:2610.20572

[62] Chenglei Shen, Haoyang Yao, Weijie Yu, Song Jin, Xiao Zhang, Jun Xu. Learning from Repaired Reasoning: Root-Cause-Guided On-Policy Distillation. 2026. arXiv preprint. arXiv:2610.03515

[63] Weichen Yu, Xiaomin Li, Yizhou Zhao, et al. Multi-Rollout On-Policy Distillation via Peer Successes and Failures. 2026. arXiv preprint. arXiv:2605.12652

[64] Jiaxin Guo, Yanwei Yue, Xuanbo Fan, Chunyu Yang, Yan Zhang. Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast. 2026. arXiv preprint. arXiv:2608.08764

[65] Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig, Timo Ropinski, Federico Tombari. On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models. 2026. arXiv preprint. arXiv:2610.32770

[66] Zhiquan Tan, Yinrong Hong. PAINT: Partial-Solution Adaptive Interpolated Training for Self-Distilled Reasoners. 2026. arXiv preprint. arXiv:2604.26573

[67] Yangrong Hu, Xueyu Zhou, Yue Wu, Jian Huang. Self-Privileged Fine-Tuning. 2026. arXiv preprint. arXiv:2611.04632

[68] Samyak Shrestha, Alexander Tessier. Rethinking Privileged Information in On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.18271

[69] Mayug Maniparambil, Ayush Rai, Ekansh Verma, Noel E. O'Connor, Fergal Reid. A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning. 2026. arXiv preprint. arXiv:2610.13744

[70] Yafeng Tang, Hao Li, Hongsheng Yu, Qiang Fu. ReTeach: Building a Self-Teacher through Multi-Round Reflection and Retry. 2026. arXiv preprint. arXiv:2610.11529

[71] Wenjie Gan, Zhaoxi Zhang, Jiahui Liang, et al. Enhancing Bug-Exposing Test Generation via Step-Level Credit Assignment and Self-Distillation from Valuable Failures. 2026. arXiv preprint. arXiv:2610.27694

[72] Haoyu Dong. Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts. 2026. arXiv preprint. arXiv:2606.10334

[73] Zenghuang Fu, Zhaoyang Li, Qiuyuan Ai, et al. UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning. 2026. arXiv preprint. arXiv:2609.34810

[74] Thomas Kleine Buening and Jonas Hübotter and Barna Pásztor and Idan Shenfeld and Giorgia Ramponi and Andreas Krause. Aligning Language Models from User Interactions. 2026. arXiv preprint. arXiv:2603.12273

[75] Xuancheng Li, Haitao Li, Yujia Zhou, et al. Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models. 2026. arXiv preprint. arXiv:2608.09109

[76] Yuwei Zhang, Sha Li, Changlong Yu, et al. Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation. 2026. arXiv preprint. arXiv:2605.12741

[77] Johnny Yu, Cyril Gorlla. How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning. 2026. arXiv preprint. arXiv:2610.36082

[78] Gaetan Narozniak, Gérard Biau, Rémi Munos, Ahmad Rammal, Pierre Marion. Distilling LLM Feedback for Lean Theorem Proving. 2026. arXiv preprint. arXiv:2605.30861

[79] Yang Li, Erik Nijkamp, Semih Yavuz, Shafiq Joty. Learning from Language Feedback via Variational Policy Distillation. 2026. arXiv preprint. arXiv:2605.15113

[80] MohammadHossein Rezaei and Anas Mahmoud and Zihao Wang and Utkarsh Tyagi and Advait Gosai and Razvan-Gabriel Dumitru and Aakash Sabharwal and Bing Liu and Yunzhong He. Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers. 2026. arXiv preprint. arXiv:2606.12507

[81] Rui Xu, Yikai Zhang, Aili Chen, Zicheng Zhao, Xu Yinghui, Libo Wu. OSPD: On-Policy Self-Distillation for Persona-Consistent Dialogue. 2026. arXiv preprint. arXiv:2609.34418

[82] Huayi Lai, Jicheng Yang, Min Yi, Chong Meng. MIRROR: From Imitation to Internalization in LLM Personalization. 2026. arXiv preprint. arXiv:2610.09795

[83] Mingfeng Lin, Chengfei Cai, Lin Xu, Chengqian Ma, Yuxiang Wei, Liang Han. PE-OPSD: Internalizing Prompt Enhancement into Flow-matching Models via On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2609.36638

[84] Ming Wen, Yuxuan Liu, Kun Yang, et al. Constitutional On-Policy Safe Distillation. 2026. arXiv preprint. arXiv:2606.03089

[85] Yingzi Ma, Zichen Zhu, Ming Jiang, Chaowei Xiao. One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context. 2026. arXiv preprint. arXiv:2607.18293

[86] Wei Fan, Qingyue Yang, Chao-Hong Tan, et al. LoOPSD: On-Policy Self-Distillation with Looped Teachers. 2026. arXiv preprint. arXiv:2610.18162

[87] Aozhe Wang, Zhengxi Lu, Jianze Wang, et al. TTPO: Test-Time Policy Optimization. 2026. arXiv preprint. arXiv:2608.27448

[88] Luke Bailey, Kaiyue Wen, Kefan Dong, Tatsunori Hashimoto, Tengyu Ma. Scaling Self-Play with Self-Guidance. 2026. arXiv preprint. arXiv:2604.20209

[89] Zhenyu Liu, Zhangquan Chen, Keyi Chen, et al. Spatial-OPSD: Self-Improving Spatial Reasoning via Label-Free Self-Distillation. 2026. arXiv preprint. arXiv:2609.37055

[90] Xiaoshu Chen, Sihang Zhou, Ke Liang, Xinwang Liu. SeOPD: Self-Evolving LLMs via Online Policy Distillation from Self-Generated Chain-of-Thought. 2026. arXiv preprint. arXiv:2609.33181

[91] Jinhao Pan, Bowen Wei, Yuqing Zhou, Ziwei Zhu. Learning from Self-Generated Responses: Reliability--Uncertainty Self-Distillation for Reasoning. 2026. arXiv preprint. arXiv:2610.19176

[92] Xiaogeng Liu, Xinyan Wang, Yingzi Ma, Yechao Zhang, Chaowei Xiao. When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning. 2026. arXiv preprint. arXiv:2605.21606

[93] Huanxin Sheng, Zhiling Ye, Haonan Wang, Jian Wang, Jinjie Gu and Jian Kang. 1% of Tokens Can Be Enough: On Gradient Estimation in On-Policy Distillation. 2026. arXiv preprint. arXiv:2609.24432

[94] Heng Liang, Xinwen Zhang, Hongchang Gao. Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models. 2026. arXiv preprint. arXiv:2610.07247

[95] Yifu Luo, Chuang Ma, Zeyu Chen, Enkelejda Kasneci, Qi Chen, Gjergji Kasneci. Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation. 2026. arXiv preprint. arXiv:2610.40574

[96] Duy Nguyen, Hanqi Xiao, Archiki Prasad, et al. AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals. 2026. arXiv preprint. arXiv:2605.20643

[97] Xingrun Xing, Haoqing Wang, Boyan Gao, Ziheng Li, Yehui Tang. Trust Region On-Policy Distillation. 2026. arXiv preprint. arXiv:2606.01249

[98] Yifei Liu, Minghao Fang, Xinyu Gu, et al. E2^2-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.05048

[99] Yugu Li, Zehong Cao, Peizhen Li, Yang Zhang, Siyi Hu, Jianglin Qiao. Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation. 2026. arXiv preprint. arXiv:2609.34848

[100] Meilin Yang, Zixuan Ding (1), Jianhao Nie, et al. Adaptive Supervised Anchoring for On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.07935

[101] Zehao Liu, Yuanpu Cao, Jinghui Chen, Vasant G. Honavar. Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning. 2026. arXiv preprint. arXiv:2605.27765

[102] Xincheng Wei, Yifan Ding, Yoshua Li, et al. Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2609.39687

[103] Qiangqiang He, Zhongheng Wu, ZiJian Wang. Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning. 2026. arXiv preprint. arXiv:2607.27888

[104] Ilgee Hong, Changlong Yu, Zhenghao Xu, et al. Training LLM Judges from Language Feedback via Position-Selective Self-Distillation. 2026. arXiv preprint. arXiv:2609.38792

[105] Seohyun Lee, Dong-Jun Han, Seyyedali Hosseinalipour, Christopher G. Brinton. Fisher-Informed Recalibration for Feedback-Based On-Policy Self-Distillation of LLMs. 2026. arXiv preprint. arXiv:2609.34009

[106] Xuan-Phi Nguyen, Zeyu Leo Liu, Yang Li, et al. Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.09263

[107] Zewen Ding, Zezhong Wu, Zhou Tao, et al. VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.28306

[108] Yangyang Feng, Zhuoyan Feng, and Junlan Chen. PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.08726

[109] Shufan Shen, Zhongni Hou, Chao Bi, Yufei Zhang, Guojun Yin, Shuhui Wang. On-Policy Self-Distillation with Teacher Evolution. 2026. arXiv preprint. arXiv:2610.38637

[110] Zile Wang, Zijian Li, Haodong Wang, et al. Teach to Learn: Hint Annealing for Self-improving LLM Reasoning. 2026. arXiv preprint. arXiv:2609.34975

[111] Zepeng Zhai, Meilin Chen. Teach Me After You Learn It: Lookahead Self-Distillation for RLVR. 2026. arXiv preprint. arXiv:2610.18256

[112] Cheng Yang, Jiayang Lyu, Shangyuan Liu, et al. iCoder-27B: Recursive AI-Led Development of Frontier Industrial Coding Model. 2026. arXiv preprint. arXiv:2609.29626

[113] Yifeng Liu, Shiyuan Zhang, Yifan Zhang, Quanquan Gu. Self-Distilled Policy Gradient. 2026. arXiv preprint. arXiv:2606.04036

[114] Ahmed Heakl, Abdelrahman M. Shaker, Youssef Mohamed, et al. CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization. 2026. arXiv preprint. arXiv:2605.19436

[115] Zhenrui Yue, Huimin Zeng, Yueqi Wang, et al. SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2609.36742

[116] ZheXu Wang, Mao-Lin Luo, Yankun Hong, et al. Outcome-Guided On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.05070

[117] Suwan Yoon, Seongryong Jung, Changhee Lee. Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization. 2026. arXiv preprint. arXiv:2610.25973

[118] Jiawei Xu, Minghui Liu, Juzheng Zhang, Tom Goldstein, Furong Huang. β\beta-OPSD: Deriving with Policy Optimization, Training with Self-Distillation. 2026. arXiv preprint. arXiv:2607.28582

[119] Xuewei Yang, Jiachen Yu, Jie Wu, Shaoning Sun, Junjie Wang, Yujiu Yang. Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning. 2026. arXiv preprint. arXiv:2606.00755

[120] Junlong Ke, Zichen Wen, Weijia Li, Conghui He, Linfeng Zhang. Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning. 2026. arXiv preprint. arXiv:2605.13255

[121] Shuning Wang, Zhiheng Wu, Xun Zhou, et al. TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models. 2026. arXiv preprint. arXiv:2609.33414

[122] Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma, et al. Does self-distillation improve reasoning?. 2026. arXiv preprint. arXiv:2610.18613

[123] Jaehoon Kim, Dongha Lee. Hindsight Compacts but Does Not Repair: Rethinking On-Policy Self-Distillation in Reasoning Models. 2026. arXiv preprint. arXiv:2605.06188

[124] Sarthak Harne, Chinmay Karkar, Yash Pandya, Ahmed Awadallah, Akshay Nambi. Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation. 2026. arXiv preprint. arXiv:2608.04794

[125] Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi, Ge Liu. The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes. 2026. arXiv preprint. arXiv:2605.11182

[126] Kaijian Zou, Emiliano Penaloza, MohammadHossein Rezaei, et al. Predicting Privileged Information Effectiveness for On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.22512

[127] Mohammadreza Armandpour, Fatih Ilhan, David Harrison, et al. Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why. 2026. arXiv preprint. arXiv:2605.10889

[128] XiuYu Zhang, Wei Chow, Junfeng Fang, Xingyu Zhu, Zhenkai Liang, Tat-Seng Chua. What Does Privileged Information Add to On-Policy Self-Distillation?. 2026. arXiv preprint. arXiv:2609.20612

[129] Ravenor Davion, Nick Rui. Privileged Context as Drift in On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.07842

[130] Keqin Peng, Chen Li, Yuanxin Ouyang, Yancheng Yuan, Liang Ding. Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2607.10805

[131] Di Huang, Hao Li, Yixin Chen, Fuhai Li. When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2609.38995

[132] Nicolas Zucchet, Scott W. Linderman. Divergence controls entropy in distillation. 2026. arXiv preprint. arXiv:2610.03529

[133] Anton Baumann, Akmal Ashirmatov, Leo Schmidt-Traub, et al. On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillation. 2026. arXiv preprint. arXiv:2609.21561

[134] Shinichi Uemura, Taiji Suzuki. Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning. 2026. arXiv preprint. arXiv:2610.05200

[135] Chen Henry Wu and Thomas Zhang and Aditi Raghunathan. Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning. 2026. arXiv preprint. arXiv:2610.05872

[136] Luis Zuin, Alexis Huet, Dario Rossi, Zied Ben Houidi. Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention. 2026. arXiv preprint. arXiv:2609.39494

[137] Md. Ismail Hossain, Humaira Kousar and Isidora Chara Tourni. Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning. 2026. arXiv preprint. arXiv:2609.37915

[138] Lei Zhao, Qichao Zhao, Bowen Zuo, Qishi Zhan. Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals. 2026. arXiv preprint. arXiv:2610.11247

[139] Karim Abdel Sadek and Federico Di Gennaro and Fanny Yang. Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation. 2026. arXiv preprint. arXiv:2610.22467

[140] Shuo Yang and Jinyang Wu and Zhengxi Lu and Yuhao Shen and Fan Zhang and Lang Feng and Shuai Zhang and Haoran Luo and Zheng Lian and Zhengqi Wen and Jianhua Tao. OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning. 2026. arXiv preprint. arXiv:2606.26790

[141] Qirui Liu, Wei Cao, Yu Mi, et al. SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning. 2026. arXiv preprint. arXiv:2610.38391

[142] Woongyeong Yeo, Yumin Choi, Taekyung Ki, Sung Ju Hwang. HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents. 2026. arXiv preprint. doi:10.18653/2026.findings-emnlp.153 arXiv:2605.17873

[143] Changle Qu, Sunhao Dai, Hengyi Cai, et al. TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning. 2026. arXiv preprint. arXiv:2608.04007

[144] Yufei Ma, Zihan Liang, Ben Chen, et al. SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning. 2026. arXiv preprint. arXiv:2605.18299

[145] Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, et al. Self-Distilled Agentic Reinforcement Learning. 2026. arXiv preprint. arXiv:2605.15155

[146] Xingyu Su, Abhishek Kumar, Qing Ping, et al. From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents. 2026. arXiv preprint. arXiv:2609.29051

[147] Fan Yang, Rui Meng, Yuxin Wen. Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?. 2026. arXiv preprint. arXiv:2607.17558

[148] Hangxi Guo, Fengyuan Liu, Yue Wang, et al. Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation. 2026. arXiv preprint. arXiv:2610.11384

[149] Yan Zhang, Daiqing Wu, Huawen Shen, et al. Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents. 2026. arXiv preprint. arXiv:2609.27307

[150] Yong Du, Tongbo Chen, Zhengxi Lu, et al. ComputerSD: Online Self-Distillation from Real-Time Feedback for Computer-Use Agents. 2026. arXiv preprint. arXiv:2609.40253

[151] Yan Zhang, Daiqing Wu, Huawen Shen, Can Ma, Yu Zhou. Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding. 2026. arXiv preprint. arXiv:2605.00642

[152] Chunji Lv, Yangguang Wei, Junlin Liu, et al. PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning. 2026. arXiv preprint. arXiv:2608.01837

[153] Xu Xia, Jinghua Piao, Min Yang, Xiaochong Lan, Jiaju Chen, Yong Li. From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents. 2026. arXiv preprint. arXiv:2607.27937

[154] Haoran Liu, Yuwei Zhang, Xiyao Li, Bohan Lyu, Jingbo Shang. HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation. 2026. arXiv preprint. arXiv:2606.11559

[155] Yuechen Luo, Zhiyi Lai, Miao Ziqi, et al. OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents. 2026. arXiv preprint. arXiv:2610.21496

[156] Sijia Li, Yuchen Huang, Zifan Liu, et al. GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation. 2026. arXiv preprint. arXiv:2605.11853

[157] Yuhao Sun, Binrui Wu, Zhuoer Xu, et al. Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation. 2026. arXiv preprint. arXiv:2609.33409

[158] Yan Yu, Zhengxi Lu, Yizhou Liu, et al. RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning. 2026. arXiv preprint. arXiv:2609.20784

[159] Tianlang Chen, Shirley Wu, Jure Leskovec. Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap. 2026. arXiv preprint. arXiv:2605.24432

[160] Azure Zhang, Ning Gao, Yuqin Dai, et al. Unlocking Proactivity in Task-Oriented Dialogue. 2026. arXiv preprint. arXiv:2605.22240

[161] Barada Sahu (Cabal AI), Shivesh Pandey (Para AI). Teach it to stop, not just to click. 2026. arXiv preprint. arXiv:2607.17136

[162] Xudong Cai, Zhengxi Lu, Yuchen Yan, et al. SWE-Ladder: Learning to Localize, Reproduce, and Patch via Progressive Privileged Self-Distillation. 2026. arXiv preprint. arXiv:2610.33667

[163] Shengjie Wang, Guanghe Li, Zonghan Yang, Yang Gao. Hindsight Hint Distillation: Scaffolded Reasoning for SWE Agents from CoT-free Answers. 2026. arXiv preprint. arXiv:2605.11556

[164] Taeckyung Lee, Rinat Amankos, Jeonghye Kim, Hyungjun Yoon, Woogyeol Jin, Sung-Ju Lee. TISD: On-Policy Self-Distillation with Trajectory Intervention. 2026. arXiv preprint. arXiv:2609.30878

[165] Wanqi Ren, Jianxiang Wang, Danxuan Liu, Linyi Ding, Huaixiao Tou. Learning from Think-Mode Advantage via On-Policy Distillation. 2026. arXiv preprint. arXiv:2609.37044

[166] Karn Tiwari, Varnith Chordia, Prathosh A P. DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation. 2026. arXiv preprint. arXiv:2610.04596

[167] Yang Li, Semih Yavuz, Shafiq Joty. RISE: Recursive Improvement via Self-Extrapolating Policy Distillation. 2026. arXiv preprint. arXiv:2609.05295

[168] Jiawei Zhou, Jianwei Wang, Chenyu Zhou, Chaojian Shi, Ming Dong, Kai Wang. EvoSQL: Memory-Augmented Critic-Generator Co-Evolution for Text-to-SQL. 2026. arXiv preprint. arXiv:2607.20489

[169] Qihui Zhu, Yuchen Wang, Zijian Wen, et al. RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models. 2026. arXiv preprint. arXiv:2607.24447

[170] Yijiang Li, Yijun Liang, Yunjie Tian, et al. Self-Supervised Visual On-Policy Distillation. 2026. arXiv preprint. arXiv:2608.14144

[171] Kaishen Wang, Dongdi Zhao, Yijun Liang, et al. Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding. 2026. arXiv preprint. arXiv:2608.25356

[172] Kanghui Tian, Siyuan Liu, Ziang Yan, Sheng Xia, Shuai Dong, Yi Wang. ViCuR: Visual Cues as Recoverable Privilege for Multimodal On-Policy Distillation. 2026. arXiv preprint. arXiv:2606.05718

[173] Yunfei Bai, Enrico Chionna, Akash Amol, Kawaljit Singh KC, Joern Tinnemeyer. K-OPSD: Verifiable On-Policy Self-Distillation for Post-Training Vision-Language Models on AEC Drawings. 2026. arXiv preprint. arXiv:2609.34082

[174] Kaixuan Du, Xin Wan, Dai Guan, Ming Chen, Xiaoxi Jiang, Guanjun Jiang. Live-OPSD: On-Policy Self-Distillation for Proactive Streaming Large Video Models. 2026. arXiv preprint. arXiv:2610.36959

[175] Yuwei Miao, Xuesheng Zhang, Wenhao Zou, et al. EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding. 2026. arXiv preprint. arXiv:2609.36803

[176] Keming Wu, Baoyi Wang, Kaichen Zhang, et al. StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for Streaming Video Understanding. 2026. arXiv preprint. arXiv:2608.16320

[177] Zhaoyu Chen, Yongwei Nie. DTP-OPSD: Reinforcement Learning with Distractor-Aware Temporal-Privilege On-Policy Self-Distillation for Temporal Video Grounding. 2026. arXiv preprint. arXiv:2611.00120

[178] Hongxing Li, Yixin Li, Dingming Li, et al. Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models. 2026. arXiv preprint. arXiv:2610.12355

[179] Haocheng Luo, Jiahui Liu, Ruicheng Zhang, et al. Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation. 2026. arXiv preprint. arXiv:2606.06076

[180] Junming Lin, Yuxuan Wang, Zhenxin Lei, et al. OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning. 2026. arXiv preprint. arXiv:2609.39490

[181] Jing Hu, Danxiang Zhu, Xianlong Luo, et al. CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillation. 2026. ACL findings 2026. arXiv:2601.16547

[182] Yujie Guo, Hongjie Chen, Jian Kang, Jie Li, Yongxiang Li, Yong Qin. TS-OPD: Reconciling ASR and QA in Speech Language Models via Task-Specific On-Policy Distillation. 2026. arXiv preprint. arXiv:2609.29464

[183] Shiao Xie, Siyu Chen, Jianwei Lv, Xiandong Li, Yujin Wang. Reinforcement Learning with Visually Attributable On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.26856

[184] Wei Zhou, Xiongwei Zhu, Lingdong Kong, et al. On-Policy Self-Distillation in Diffusion Models. 2026. arXiv preprint. arXiv:2608.24646

[185] Hongyu Liu, Chun Wang, Feng Gao, et al. OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators. 2026. arXiv preprint. arXiv:2607.08766

[186] Shiyi Zhang, Mushui Liu, Yunze Tong, et al. Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher. 2026. arXiv preprint. arXiv:2608.26872

[187] Yifu Luo, Zeyu Chen, Haoyu Wang, et al. Learning from the Self-future: On-policy Self-distillation for dLLMs. 2026. arXiv preprint. arXiv:2606.18195

[188] Hyeongsoo Lim, Jinyoung Kim, Eunseo Seo, Minho Jang, Jiwon Yoon. SelFusion: Self-distillation for Diffusion Language Models. 2026. ACL 2026. doi:10.18653/v1/2026.acl-long.1008 arXiv:2608.22898

[189] Xiaohang Tang, Keyue Jiang, Che Liu, et al. GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models. 2026. arXiv preprint. arXiv:2605.29398

[190] Liangbing Zhao, Le Zhuo, Mohamed Elhoseiny. On-Policy Self-Distillation for Multi-Turn Image Editing. 2026. arXiv preprint. arXiv:2609.35611

[191] Shengxiang Ji, Boyang Wang, Haiyang Xu, et al. LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2609.38146

[192] Shafeef Omar, Sebastian Sanokowski, Elham Daneshmand, et al. DiPLO: Self-distilled Diffusion Planning for Loco-manipulation. 2026. arXiv preprint. arXiv:2610.24847

[193] Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber, et al. iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs. 2026. arXiv preprint. arXiv:2609.24646

[194] Jiarui Liu, Lechen Zhang, Yongjin Yang, et al. MixSD: Mixed Contextual Self-Distillation for Knowledge Injection. 2026. arXiv preprint. arXiv:2605.16865

[195] Shuaike Li, Kai Zhang, Xianquan Wang, Jiachen Liu, Shengpeng Mo. From Fact Overwriting to Knowledge Evolution: Causal Editing via On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2605.28303

[196] Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, Furu Wei. Online Experiential Learning for Language Models. 2026. arXiv preprint. arXiv:2603.16856

[197] Yuhang Zhou, Yifan Wu, Bo Peng, et al. Self-Evolving Agents through Privileged Retrospective Distillation. 2026. arXiv preprint. arXiv:2610.27495

[198] Shijie Ren, Xiting Wang, Meng Li, et al. Self-Improving Large Language Models via Progressive Experience Evolution. 2026. arXiv preprint. arXiv:2608.02139

[199] Yiqiao Qiu. Replay for LLM Continual Learning with Multi-Teacher On-Policy Distillation. 2026. arXiv preprint. doi:10.5281/zenodo.23906641 arXiv:2611.02746

[200] Yifei Li, Rongman Xu, Lingling Zhang, et al. Self-Specialized Teachers for Domain Post-Training. 2026. arXiv preprint. arXiv:2608.28647

[201] Dian Jin, Jinluan Yang, Kairong Han, et al. MDSKILL: Learning to Internalize Skill Memory through Multi-View Distillation for Efficient Dynamic Agent Self-Evolution. 2026. arXiv preprint. arXiv:2611.05313

[202] Hojun Cho, Kang Eun Jeon, Jaegul Choo. Adaptive-Policy Distillation: Knowledge Injection under Information Asymmetry. 2026. arXiv preprint. arXiv:2611.00852

[203] Ruiyang Qin, Qingzhuo Wang, Dongrui Liu, Qiang Li, Zhihua Wei, Wen Shen. Multilingual Safety Alignment via Self-Distillation. 2026. arXiv preprint. arXiv:2605.02971

[204] Shuo Hong, Jiaxin Ma, Xidi Cai, Qianli Ma. Erase the Injected Region: Multimodal Prompt-Injection Defense with On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2610.41307

[205] Yilun Qiu, Xiaoyan Zhao, Chengbing Wang, et al. Rubric-Aware On-Policy Self-Distillation for LLM Personalization. 2026. arXiv preprint. arXiv:2609.35262

[206] Dingyao Yu, Tong Zhang, Yutao Mou, Yunxiao Zhang, Wei Ye, Shikun Zhang. Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2608.14684

[207] Mingxuan Xia, Yuhang Yang, Chao Ye, et al. CriPO: Enhancing Rubric-based RL via Self-Distillation. 2026. arXiv preprint. arXiv:2607.18082

[208] Chengsheng Zhang and Wenshen Xu and Yuchuan Wu and Zhiyuan Yu and Junfeng Fang and Tianlong Wang and Bingxue Qiu and Yongbin Li and Xiang Wang. Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation. 2026. arXiv preprint. arXiv:2610.38566

[209] Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Qinyuan Ye, Caiming Xiong, Chien-Sheng Wu. The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation. 2026. arXiv preprint. arXiv:2604.16830

[210] Tao Feng and Xinke Jiang and Chao Wu. KbSD: Knowledge Boundary aware Self-Distillation for Behavioral Calibration in Agentic Search. 2026. arXiv preprint. arXiv:2606.29863

[211] Mohammad Sadegh Akhondzadeh, Soroush H. Zargarbashi, Aleksandar Bojchevski. Adversarial Self-Distillation. 2026. arXiv preprint. arXiv:2610.38732

[212] Chang Wu, Junfeng Fang, Houcheng Jiang, et al. PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models. 2026. arXiv preprint. arXiv:2606.25442

[213] Durgesh Kalwar, Vardhan Palod, Jaya Adithya Pavuluri, Subbarao Kambhampati. Masked Self-Distillation: Internalizing the Chain-of-Thought in Language Models. 2026. arXiv preprint. arXiv:2607.22629

[214] Yishuo Cai, Jiahui Liu, Yuanxin Liu, et al. Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillation. 2026. arXiv preprint. arXiv:2606.08719

[215] Chenjun Xu, Zhennan Zhou, Zhan Su, Bill Howe, Lucy Lu Wang, Bingbing Wen. STOP: Structured On-Policy Pruning of Long-Form Reasoning in Low-Data Regimes. 2026. arXiv preprint. arXiv:2605.13165

[216] Junxian Li, Ruixuan Yang, Tianao Zhang, Tiange Xu, Weisheng Dong, Yulun Zhang. Fewer Tokens, More Self-Teaching: On-Policy Self-Distillation for Extreme Visual Token Reduction. 2026. arXiv preprint. arXiv:2609.32353

[217] Ahmadreza Jeddi, Enming Zhang, Jasper Gerigk, et al. SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models. 2026. arXiv preprint. arXiv:2609.34044

[218] John Kirchenbauer, Abhimanyu Hans, Brian Bartoldson, Micah Goldblum, Ashwinee Panda, Tom Goldstein. Multi-Token Prediction via Self-Distillation. 2026. arXiv preprint. arXiv:2602.06019

[219] Shuiyuan Wang, Bingshen Mu, Pengshen Zhang, et al. On-Policy Self-Distillation for Multi-Dialect ASR: Mastering Dialects, Retaining Mandarin. 2026. arXiv preprint. arXiv:2608.11898

[220] Rongfeng Guo, Zhuang Zhuang, Ji Dai, et al. Search-Aware On-Policy Self-Distillation with Sibling Alignment for Generative Recommendation. 2026. arXiv preprint. arXiv:2611.01568

[221] Rui Zhu and Minglong Cao and Chenyu Zhou and Jianghao Lin and Dongdong Ge. Beyond Verified Answers: Solver-Informed Self-Distillation for Bootstrapping Operations Research Language Models. 2026. arXiv preprint. arXiv:2609.09957

[222] Kejing Wang, Toan Nguyen, Minh Hoang Nguyen, Simon Khan, Flora D. Salim. ROAD-VLA: Robust Online Adaptation via Self-Distillation for Vision-Language-Action Models. 2026. arXiv preprint. arXiv:2606.25800

[223] Yuechen Luo, He Chong, Fang Li, et al. Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving. 2026. arXiv preprint. arXiv:2610.13453

[224] Hao Dou. FIRE-VLA: Failure-Informed Self-Evolution for Vision-Language-Action Models in Autonomous Driving. 2026. arXiv preprint. arXiv:2608.13395

[225] Bowen Feng, Keqin Wang, Yulong Yang, et al. Training Generalist Navigation Policies via Action Self-Distillation. 2026. arXiv preprint. arXiv:2610.17593

[226] Keyu Chen, Nanfei Ye, Yida Wang, et al. CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies. 2026. arXiv preprint. arXiv:2605.04470

[227] Dongyue Li and Zebang Cheng and Zhongyi Deng and Hongboxu and Yi Bin and Fei Ma. RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport. 2026. arXiv preprint. arXiv:2610.37281

[228] Mingzhou Jiang, Peixi Wu, Hang Cheng, et al. Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings. 2026. arXiv preprint. arXiv:2609.15296

[229] Minye Shao, Chaohui Yu, Yixuan Wu, et al. From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoning. 2026. arXiv preprint. arXiv:2609.39566

[230] Yuhan Wang, Zhengxi Lu, Yuchen Yan, et al. PaperGym: Rubric-Centered Evolution for Research-Plan Generation. 2026. arXiv preprint. arXiv:2608.31119

[231] Jie Sun, Mao Zheng, Mingyang Song, et al. EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models. 2026. arXiv preprint. arXiv:2607.11012

[232] Wonjun Kang, Kevin Galim, Seunghyuk Oh, et al. AsyncOPD: How Stale Can On-Policy Distillation Be?. 2026. arXiv preprint. arXiv:2606.24143

[233] Xianwei Chen, Shimin Zhang, Jibin Wu. f\boldsymbol{f}-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control. 2026. arXiv preprint. arXiv:2605.17862

[234] Dongyang Chen, Shudong Liu, Shubo Li, et al. Branching On-Policy Distillation. 2026. arXiv preprint. arXiv:2610.13481

[235] Shangzhe Li, Yuxiao Yang, Tianrun Yu, et al. An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning. 2026. arXiv preprint. arXiv:2609.35505

[236] Zihan Chen, Hengguang Zhou, Yuan Kang, et al. BAS-OPD: Budget-Aware Selective On-Policy Self-Distillation for Fine-Grained Multimodal Perception. 2026. arXiv preprint. arXiv:2609.25891

[237] Qirui Liu, Yichen Sun, Yan Wang, et al. EOPSA: Efficient On-Policy Self-Distilled Safety Alignment. 2026. arXiv preprint. arXiv:2609.34519

Cite this report

bibtex
@misc{parallax-2026-on-policy-self-distillation-survey, title = {{On-Policy Self-Distillation for Large Language Models: A Survey}}, author = {{Parallax}}, year = {2026}, month = nov, howpublished = {\url{https://parallax.ink/research/en/on-policy-self-distillation-survey}}, url = {https://parallax.ink/research/en/on-policy-self-distillation-survey}, note = {Parallax Research, Survey} }

Keep reading