Language models can explain neurons in language models | OpenAI Read paper(opens in a new window)View neurons(opens in a new window)View code and dataset(opens in a new window) We use GPT‑4 to automatically write explanations for the behavior of neurons in large language models and to score those explanations. We release a dataset of these (imperfect) explanations and scores for every neuron in GPT‑2.
核心贡献 · Key contributions
提出使用 GPT-4 自动生成并评分大语言模型中神经元的自然语言解释。 Proposes using GPT-4 to automatically generate and score natural language explanations for neurons in LLMs.
发布了 GPT-2 全部 307,200 个神经元的解释与评分数据集。 Releases a dataset of explanations and scores for all 307,200 neurons in GPT-2.
证明解释质量随解释模型规模增大和迭代优化而提升。 Demonstrates that explanation quality improves with larger explainer models and iterative refinement.
发现超过 1000 个高分解释（评分≥0.8）的神经元，但多数不具趣味性。 Finds over 1,000 neurons with high-scoring explanations (score ≥ 0.8), though most are not interesting.
表明改变被解释模型的激活函数可提升解释评分。 Shows that changing activation functions in the explained model can improve explanation scores.
开源代码和可视化工具，以促进社区对神经元可解释性的研究。 Open-sources code and visualization tools to enable community research on neuron interpretability.
局限 · Limitations
解释限于简短自然语言，无法捕捉复杂的多语义行为。 Explanations are limited to short natural language, unable to capture complex polysemantic behavior.
方法仅解释神经元对输入的反应，忽略下游效应和回路。 Method only explains neuron behavior as a function of input, ignoring downstream effects and circuits.
高分解释可能因相关性本质而在分布外文本上失效。 High-scoring explanations may still fail on out-of-distribution texts due to correlational nature.
整体流程计算密集，限制了向超大模型的扩展性。 The overall procedure is compute-intensive, limiting scalability to very large models.
即使 GPT-4 的解释也逊于人工解释，表明仍有很大改进空间。 Even GPT-4 explanations are worse than human-written ones, indicating significant room for improvement.
论文章节 · Sections（共 8）
概述Overview
工作原理How it works
步骤 1：使用 GPT-4 生成解释Step 1: Generate explanation using GPT-4