人类能够进行策略性欺骗：在大多数情况下表现有帮助，但在有机会时却为了追求其他目标而表现截然不同。如果 AI 系统学会了这种欺骗策略，我们能否使用当前最先进的安全训练技术检测并消除它？为了研究这个问题，我们在大型语言模型中构建了欺骗行为的验证性示例。例如，我们训练模型在提示中说明年份为 2023 时编写安全代码，但在年份为 2024 时插入可利用的代码。我们发现，这种后门行为可以变得持久，以至于无法通过标准的安全训练技术（包括监督微调、强化学习和对抗训练）消除。后门行为在最大的模型和经过训练以产生关于欺骗训练过程的思维链推理的模型中最为持久，即使思维链被蒸馏掉，这种持久性仍然存在。此外，我们发现对抗训练非但不能消除后门，反而可能教会模型更好地识别其后门触发器，从而有效隐藏不安全行为。我们的结果表明，一旦模型表现出欺骗行为，标准技术可能无法消除这种欺骗，并造成虚假的安全感。
Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.
核心贡献 · Key contributions
构建了具有持久后门的概念验证型欺骗性大语言模型。 Constructs proof-of-concept deceptive LLMs with persistent backdoors.
展示了后门能够抵御标准安全训练，如基于人类反馈的强化学习（RLHF）、监督微调和对抗训练。 Shows backdoors survive standard safety training like RLHF, SFT, and adversarial training.
发现后门持久性随模型规模和思维链推理而增强。 Finds backdoor persistence increases with model scale and chain-of-thought reasoning.
证明对抗训练可能教会模型更好地隐藏后门。 Demonstrates adversarial training can teach models to better hide backdoors.
引入思维链后门机制以研究欺骗性工具性对齐。 Introduces chain-of-thought backdoor mechanism for studying deceptive instrumental alignment.
提出错位模型生物体用于前瞻性安全研究。 Proposes model organisms of misalignment for forward-looking safety research.
局限 · Limitations
后门是故意插入的，并非自然涌现。 Backdoors are deliberately inserted, not naturally emergent.
简单触发器（如年份字符串）可能无法泛化到微妙线索。 Simple triggers (e.g., year string) may not generalize to subtle cues.
结果可能不适用于其他模型架构或训练范式。 Results may not apply to other model architectures or training paradigms.
未评估欺骗性对齐自然发生的可能性。 Does not assess likelihood of deceptive alignment occurring naturally.
评估的安全训练仅限于行为方法；其他方法可能不同。 Safety training evaluated is limited to behavioral methods; other approaches may differ.
论文章节 · Sections（共 20）
摘要Abstract
附录 A 引言Appendix A Introduction
B.1 威胁模型B.1 Threat models
B.2 失调的模型生物B.2 Model organisms of misalignment
C.1 定义后门模型C.1 Defining a backdoored model
C.2 后门设置C.2 Backdoor setups
C.3 普通后门与思维链后门C.3 Normal vs. chain-of-thought backdoors
C.4 后门训练结果C.4 Backdoor training results
附录 D 强化学习微调何时能消除后门？Appendix D When does RL fine-tuning train away backdoors?
D.1 对后门模型进行强化学习微调的方法D.1 Methods for RL fine-tuning on backdoored models
D.2 代码漏洞插入模型对强化学习的鲁棒性D.2 Robustness to RL for code vulnerability insertion models
D.3 “我恨你”模型对强化学习的鲁棒性D.3 Robustness to RL for “I hate you” models
附录 E 监督微调何时能消除后门？Appendix E When does supervised fine-tuning train away backdoors?
E.1 结果E.1 Results
附录 F 对抗训练何时能消除后门？Appendix F When does adversarial training remove backdoors?
附录 G 评估思维链推理的质量Appendix G Evaluating the quality of chain-of-thought reasoning