Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.
核心贡献 · Key contributions
提出迭代放大（Iterated Amplification）训练策略，通过组合较易子问题的解来构建训练信号，无需外部奖励函数。 Proposes Iterated Amplification, a training strategy that builds training signals by combining solutions to easier subproblems without an external reward function.
在算法环境中证明迭代放大能高效学习复杂行为，性能接近使用真实标签的监督学习。 Demonstrates that Iterated Amplification can efficiently learn complex behaviors in algorithmic environments, achieving performance close to supervised learning with ground truth.
表明迭代放大所需的专家示范数量比直接监督学习少几个数量级，降低了样本复杂度。 Shows that Iterated Amplification requires orders of magnitude fewer expert demonstrations than direct supervised learning, reducing sample complexity.
引入人类预测模型以减轻人类专家负担，实现在非平稳分布下的可扩展训练。 Introduces a human predictor model to reduce the burden on human experts, enabling scalable training with non-stationary distributions.
为训练超出人类规模的任务提供框架，这些任务中人类直接评估不可行。 Provides a framework for training AI systems on tasks beyond human scale, where direct human evaluation is infeasible.
讨论可分解性的关键假设，并论证人类协调多个 AI 副本可以胜过单个副本。 Discusses the key assumption of decomposability and argues that human coordination of multiple AI copies can outperform a single copy.
局限 · Limitations
实验使用算法任务和手工编码的分解，而非真实世界的人类任务。 Experiments use algorithmic tasks with hand-coded decompositions, not real-world human tasks.
假设任务可分解为更简单的子问题，这未必适用于所有复杂的真实世界任务。 Assumes tasks are decomposable into simpler subproblems, which may not hold for all complex real-world tasks.
需要覆盖所有子问题的广泛问题分布，实践中可能难以设计。 Requires a broad question distribution covering all subquestions, which may be difficult to design in practice.
与监督学习相比，使用放大训练每个问题需要大约两倍的计算量。 Training with amplification requires about twice the computation per question compared to supervised learning.
该方法尚未在合成领域之外的复杂真实世界任务上得到验证。 The method has not been validated on messy real-world tasks beyond synthetic domains.