Neural networks often pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in "superposition." We demonstrate the existence of a phase change, a surprising connection to the geometry of uniform polytopes, and evidence of a link to adversarial examples. We also discuss potential implications for mechanistic interpretability.
核心贡献 · Key contributions
在玩具 ReLU 网络中演示叠加现象，通过容忍干扰实现特征数超过维度数。 Demonstrates superposition in toy ReLU networks, enabling more features than dimensions by tolerating interference.
揭示叠加中的相变，基于稀疏性和重要性的表示策略之间存在尖锐转变。 Reveals a phase change in superposition, with sharp transitions between representational strategies based on sparsity and importance.
将叠加几何与均匀多面体（如四面体、五边形）联系起来，关联到汤姆孙问题。 Connects superposition geometry to uniform polytopes (e.g., tetrahedrons, pentagons), linking to the Thomson problem.
发现叠加与对抗样本关联的证据，暗示共享机制。 Finds evidence linking superposition to adversarial examples, suggesting a shared mechanism.
提出叠加解释多义性，对机制可解释性有启示。 Proposes that superposition explains polysemanticity and has implications for mechanistic interpretability.
局限 · Limitations
结果来自合成稀疏特征的玩具模型，推广到真实神经网络尚不明确。 Results are from toy models with synthetic sparse features; generalization to real neural networks is unclear.
均匀叠加假设（重要性及稀疏性均等）不现实，真实特征是非均匀的。 Uniform superposition assumption (equal importance and sparsity) is unrealistic; real features exhibit non-uniformity.
相变演示依赖理想条件，真实模型可能具有更平滑的转变。 Phase change demonstration relies on idealized conditions; real models may have smoother transitions.
全篇假设的线性表示假说存在争议，可能并非在所有情况下成立。 Linear representation hypothesis, assumed throughout, is debated and may not hold in all cases.
与对抗样本的关联是初步的，缺乏严格的理论证明。 Link to adversarial examples is preliminary, lacking rigorous theoretical proof.
论文章节 · Sections（共 24）
摘要Abstract
1 深入探讨：电路简介1 Zoom In: An Introduction to Circuits
2 Softmax 线性单元2 Softmax Linear Units
3 压缩感知3 Compressed sensing
4 局部编码与分布式编码4 Local vs. Distributed Coding
6 Curve Detectors6 Curve Detectors [link]
7 多个模型叠加为一个模型7 Superposition of many models into one
13 深度场景 CNN 中的目标检测器涌现13 Object detectors emerge in deep scene cnns [PDF]
18 高低频检测器18 High-Low Frequency Detectors
19 人工神经网络中的多模态神经元19 Multimodal Neurons in Artificial Neural Networks
21 特征可视化 [链接]21 Feature Visualization [link]
证明与反驳23 Proofs and refutations
25 通过线性规划解码25 Decoding by linear programming
27 上下文学习和归纳头27 In-context Learning and Induction Heads [HTML]
33 对抗性球体33 Adversarial spheres
可解释性的构建模块37 The Building Blocks of Interpretability [link]
38 可视化权重38 Visualizing Weights [link]
41 InceptionV1 早期视觉概述41 An Overview of Early Vision in InceptionV1
44 通过因子分解解耦44 Disentangling by factorising
压缩感知与最佳项逼近46 Compressed sensing and best -term approximation
稀疏恢复的下界49 Lower bounds for sparse recovery
51 压缩感知：RIP 有多锐利51 Compressed sensing: How sharp is the RIP
54 使用生成模型的压缩感知54 Compressed Sensing using Generative Models [HTML]