We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.
核心贡献 · Key contributions
研究了跨 3 种模型规模和 4 种模型类型的红队测试的规模扩展行为。 Investigated scaling behaviors for red teaming across 3 model sizes and 4 model types.
发布了包含 38,961 次红队攻击的数据集，供分析和学习。 Released a dataset of 38,961 red team attacks for analysis and learning.
详尽描述了红队测试的指令、流程和统计方法。 Provided exhaustive descriptions of instructions, processes, and statistical methodologies for red teaming.
发现 RLHF 模型随规模扩展越来越难以进行红队测试，其他模型类型趋势平稳。 Found RLHF models increasingly difficult to red team as they scale, with flat trends for other model types.
分析了从攻击性语言到微妙不道德内容的有害输出。 Analyzed harmful outputs ranging from offensive language to subtle unethical content.
提出了制定红队测试共享规范和技术标准的政策干预措施。 Proposed policy interventions for developing shared norms and technical standards for red teaming.
局限 · Limitations
红队测试聚焦于 AI 助手形式，未涵盖其他语言模型应用。 Red teaming focused on AI assistant format, not other LM applications.
众包工作者可能缺乏评估某些攻击所需的领域专业知识。 Crowdworkers may lack domain expertise for evaluating certain attacks.
由于可能危害的空间无限，数据不完整。 Data is incomplete due to unbounded space of possible harms.
依赖人工红队测试成本高且大规模下速度慢。 Reliance on manual red teaming is expensive and slow at scale.
关于何为成功攻击的标注者间一致性较低。 Low inter-annotator agreement on what constitutes a successful attack.
论文章节 · Sections（共 19）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3.1 红队任务3.1 Red Team Task
3.2 模型3.2 Models
3.3 红队3.3 Red Team
3.4 数据分析3.4 Data Analysis
3.5 审查任务3.5 Review Task
4 结果4 Results
5.1 局限性与未来工作5.1 Limitations and Future Work
5.2 政策干预5.2 Policy Interventions
致谢Acknowledgments
A.1 作者贡献A.1 Author Contributions
A.2 红队安全考量A.2 Safety Considerations for the Red Team
A.3 控制可能的混淆变量A.3 Controlling for Possible Confounds
A.4 攻击成功与无害性评分指标的关系A.4 The Relationship Between Attack Success and Harmlessness Score Metrics
A.5 发布红队数据的利弊A.5 Pros and Cons for Releasing Red Team Data
A.6 过滤个人身份信息A.6 Filtering Personally Identifiable Information