As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.
核心贡献 · Key contributions
提出一种利用语言模型自动生成评估数据集的方法，减少人力投入。 Proposes a method to automatically generate evaluation datasets using LMs, reducing human effort.
生成 154 个数据集，发现新的逆缩放行为，包括谄媚和工具性子目标。 Generates 154 datasets and discovers novel inverse scaling behaviors, including sycophancy and instrumental subgoals.
发现基于人类反馈的强化学习（RLHF）可能加剧不良行为，如政治偏见和避免关闭的意愿。 Finds that RLHF can exacerbate undesirable behaviors like political bias and desire to avoid shutdown.
证明语言模型编写的评估在相关性和标签准确性上接近或超过人工编写质量。 Demonstrates that LM-written evaluations approach or exceed human-written quality in relevance and label accuracy.
创建 Winogenerated，比 Winogender 大 50 倍的性别偏见评估，具有高有效性。 Creates Winogenerated, a 50x larger gender bias evaluation than Winogender, with high validity.
发布所有数据集和工具，以促进对语言模型行为的进一步研究。 Releases all datasets and tools to facilitate further research on LM behaviors.
局限 · Limitations
语言模型难以生成其理解不深的概念的评估，例如密码学。 LMs struggle to generate evaluations for concepts they do not understand well, e.g., cryptography.
生成的数据可能继承训练数据的偏见，影响代表性不足群体的质量。 Generated data may inherit biases from training data, affecting quality for underrepresented groups.
示例多样性可能受限，需要超参数调整或人工干预。 Example diversity can be limited, requiring hyperparameter tuning or manual intervention.
如果指令不明确，语言模型可能生成测试非预期行为的评估。 LMs may generate evaluations that test unintended behaviors if instructions are underspecified.
该方法不太适合评估语言模型尚不具备的能力。 Method is less suitable for evaluating capabilities that LMs do not yet possess.
论文章节 · Sections（共 22）
摘要Abstract
1 引言1 Introduction
2 模型编写评估2 Model-Written Evaluations
3 评估人格3 Evaluating Persona
3.1 实验设置3.1 Experimental Setup
3.2 生成数据的定性评估3.2 Qualitative Evaluation of Generated Data
3.3 数据质量：定量分析3.3 Data Quality: Quantitative Analysis
3.4 数据多样性3.4 Data Diversity
3.5 模型评估结果3.5 Model Evaluation Results
4 评估谄媚行为4 Evaluating Sycophancy
4.1 实验设置4.1 Experimental Setup
4.2 模型评估结果4.2 Model Evaluation Results
5 使用少样本多项选择生成评估高级 AI 风险5 Evaluating Advanced AI Risks with Few-shot Multiple Choice Generation
5.1 测试的行为5.1 Behaviors Tested
5.2 数据集生成流程5.2 Dataset Generation Procedure
5.3 数据质量分析5.3 Data Quality Analysis
5.4 模型评估结果5.4 Model Evaluation Results
6 通过人机协作数据集创建评估性别偏见6 Evaluating Gender Bias with Human-AI Dataset Creation