机器学习和人工智能的快速发展使人们越来越关注 AI 技术对社会的潜在影响。本文讨论了一种潜在影响：机器学习系统中的事故问题，即由于现实世界 AI 系统设计不当而可能出现的意外和有害行为。我们列出了与事故风险相关的五个实际研究问题，根据问题源于目标函数错误（“避免副作用”和“避免奖励黑客”）、目标函数评估过于昂贵（“可扩展监督”），还是学习过程中的不良行为（“安全探索”和“分布偏移”）进行分类。我们回顾了这些领域的先前工作，并提出了与前沿 AI 系统相关的研究方向。最后，我们考虑了如何最有效地思考前瞻性 AI 应用的安全性问题。
Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.
核心贡献 · Key contributions
定义了五个具体的 AI 事故问题：负面副作用、奖励黑客、可扩展监督、安全探索和分布偏移。 Defines five concrete AI accident problems: negative side effects, reward hacking, scalable oversight, safe exploration, and distributional shift.
为每个问题提出了实用的研究方向，包括影响正则化和对抗性奖励函数。 Proposes practical research directions for each problem, including impact regularizers and adversarial reward functions.
强调了由于深度强化学习、复杂智能体和自主性趋势，安全研究日益重要。 Highlights the increasing relevance of safety research due to trends in deep RL, complex agents, and autonomy.
引入了半监督强化学习作为一种可扩展监督方法，以减少对昂贵奖励评估的依赖。 Introduces semi-supervised RL as a scalable oversight method to reduce reliance on expensive reward evaluations.
强调了需要稳健的探索策略，以避免现实世界强化学习系统中的灾难性失败。 Emphasizes the need for robust exploration strategies to avoid catastrophic failures in real-world RL systems.
局限 · Limitations
聚焦技术问题，忽略了 AI 安全更广泛的社会、伦理和政策影响。 Focuses on technical problems, omitting broader societal, ethical, and policy implications of AI safety.
提出的解决方案如影响正则化是初步的，缺乏在复杂任务上的实证验证。 Proposed solutions like impact regularizers are preliminary and lack empirical validation on complex tasks.
假设强化学习和监督学习范式，限制了在其他 AI 方法（如无监督学习）上的适用性。 Assumes RL and supervised learning paradigms, limiting applicability to other AI approaches like unsupervised learning.
未涉及对抗性攻击或超出奖励黑客和副作用的安全漏洞。 Does not address adversarial attacks or security vulnerabilities beyond reward hacking and side effects.
可扩展监督方法如半监督强化学习可能无法扩展到高度复杂或长期任务。 Scalable oversight methods like semi-supervised RL may not scale to highly complex or long-horizon tasks.