Lightning OPD 2.0：缓解大型推理模型跨教师在线策略蒸馏中的风格偏差

Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models

韩松 Song Han · · 2026-07-30 · arXiv:2607.28449 ↗

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

在线策略蒸馏（OPD）从教师模型提供密集的令牌级监督，但其有效性可能依赖于教师一致性，即提供 OPD 监督的模型也应生成用于训练监督微调（SFT）参考的演示数据。然而，在实践中，当 SFT 数据来源混合或未知，或当 SFT 数据生成和后续蒸馏偏好使用不同模型时，这一条件经常被违反。在这种跨教师设置下，即使更强的 OPD 教师也可能相比 SFT 参考几乎没有改进。我们发现，原始教师-参考分歧既包含可能有用的上下文特定教师证据，也包含一个与措辞、格式和推理节奏差异相关的重复组成部分。我们引入了带有交叉拟合风格残差化的 Lightning OPD 2.0，该方法使用 rollout 级交叉拟合来估计这一重复组成部分，作为风格令牌偏差的操作性代理，并在构建令牌级 OPD 更新之前将其减去。在数学推理和代码生成基准测试中，Lightning OPD 2.0 在跨教师设置下 consistently 优于 Lightning OPD。从 Klear-Reasoner-8B-SFT 出发，Lightning OPD 2.0 在 AIME 2024 上达到 82.4%，在 LiveCodeBench v5 上达到 63.0%。这些结果共同确立了 Lightning OPD 2.0 作为跨教师 OPD 的实用方法，放宽了教师一致性作为先决条件，并允许 SFT 数据生成器和蒸馏教师独立选择。代码即将发布。

On-policy distillation (OPD) provides dense token-level supervision from a teacher, but its effectiveness can depend on teacher consistency, meaning that the model providing OPD supervision should also have generated the demonstrations used to train the supervised fine-tuning (SFT) reference. However, this condition is frequently violated in practice when SFT data have mixed or unknown provenance or when different models are preferred for SFT data generation and subsequent distillation. In such cross-teacher settings, even a stronger OPD teacher can yield little improvement over the SFT reference. We find that raw teacher--reference disagreement contains potentially useful context-specific teacher evidence as well as a recurring component associated with differences in wording, formatting, and reasoning cadence. We introduce Lightning OPD 2.0 with cross-fitted style residualization, which uses rollout-level cross-fitting to estimate this recurring component as an operational proxy for style-token bias and subtracts it before constructing the token-level OPD update. Across mathematical reasoning and code generation benchmarks, Lightning OPD 2.0 consistently outperforms Lightning OPD in cross-teacher settings. Starting from Klear-Reasoner-8B-SFT, Lightning OPD 2.0 reaches 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5. Together, these results establish Lightning OPD 2.0 as a practical approach to cross-teacher OPD, relaxing teacher consistency as a prerequisite and allowing the SFT data generator and distillation teacher to be selected independently. Code will be released soon.

核心贡献 · Key contributions

• 提出跨教师在线策略蒸馏（cross-teacher OPD）设置：SFT 数据生成器与 OPD 教师独立选择，并证明该设置会带来显著的蒸馏性能下降。
  Formulates the cross-teacher on-policy distillation setting where the SFT data generator and OPD teacher are chosen independently, showing substantial performance degradation.
• 识别出教师-参考模型分歧中可跨 rollout 预测的重复成分，并将其与风格 token 偏差相关联，解释原始跨教师监督为何会产生误导。
  Identifies a recurring cross-rollout component in teacher–reference disagreement tied to style-token bias, explaining why raw cross-teacher supervision can mislead.
• 提出 Lightning OPD 2.0 的交叉拟合风格残差化：在 token 级 OPD 更新前，用 token 身份与上下文查找表估计并减去重复的风格偏差。
  Introduces Lightning OPD 2.0 cross-fitted style residualization, estimating recurring style bias via token/context lookup tables and subtracting it before token-level OPD updates.
• 在两个跨教师设置中，于数学推理与代码生成基准上超越 Lightning OPD，在 AIME 2024 上达到 82.4%，在 LiveCodeBench v5 上达到 63.0%。
  Outperforms Lightning OPD in two cross-teacher settings on math and code benchmarks, reaching 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5.
• 放宽教师一致性作为有效 OPD 的前提要求，使 SFT 数据生成器与蒸馏教师的选择解耦。
  Relaxes teacher consistency as an OPD prerequisite, decoupling SFT data generator and distillation teacher selection.

局限 · Limitations

• 评估仅限于数学/代码任务与 Qwen 系列模型；尚未验证对其他领域或模型族的泛化性。
  Evaluation is limited to math/code tasks and Qwen-family models; generalization to other domains or model families is not established.
• 风格/推理分解是操作性而非真实语义划分；可预测成分可能包含非风格因素，残差也可能保留风格噪声。
  The style/reasoning decomposition is operational, not a ground-truth semantic partition; predictable effects may include non-style factors and residuals may retain style noise.
• 要求教师与参考模型具备兼容的 tokenization，才能对齐 token 级对数概率，限制了在 tokenizer 不兼容模型上的应用。
  Requires compatible tokenization between teacher and reference for aligned token-level log-probabilities, limiting application across tokenizer-incompatible models.
• 该方法针对 Lightning OPD 的冻结离线回放开发；是否能直接用于完全在线的跨教师蒸馏尚未经实验验证。
  Developed for the frozen offline replay of Lightning OPD; direct applicability to fully online cross-teacher distillation is not empirically tested.
• 仅测试了两个跨教师设置和一个 OPD 教师；Klear 设置使用诊断代理，而非精确的教师一致信号。
  Only two cross-teacher settings and one OPD teacher are tested; Klear uses a diagnostic proxy, not an exact teacher-consistent signal.

论文章节 · Sections（共 17）

• 摘要 Abstract
• Lightning OPD 2.0：大型推理模型跨教师在线蒸馏中的风格偏差缓解 Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
• 1 引言 1 Introduction
• 大语言模型后训练 LLM Post-Training.
• 在策略蒸馏 On-Policy Distillation.
• 3.1 跨教师 OPD 3.1 Cross-Teacher OPD
• 3.2 跨 rollout 的可预测性作为风格代理 3.2 Predictability across Rollouts as a Style Proxy
• 反复出现分歧的可观测坐标 Observable coordinates for recurring disagreement.
• 3.3 重复分歧的交叉拟合估计 3.3 Cross-Fitted Estimation of Recurring Disagreement
• 3.4 残差化 Lightning OPD 3.4 Residualized Lightning OPD
• 4 实验 4 Experiments
• 跨教师设置 Cross-teacher settings.
• 训练设置 Training Settings.
• 评估 Evaluation.
• 基线 Baselines.
• 4.3 风格标记偏差的机制分析 4.3 Mechanism Analysis of Style-Token Bias
• 5 结论 5 Conclusion

阅读逐段中英对照全文 →