I Can See Everything
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文考察了 Nick Cammarata 的观点，即神经网络已经“干净”，也就是说，较大 LLM 中的大多数神经元表征的是单一概念，而不是通过叠加（superposition）承载互不相关的概念。作者认为，尽管这项工作是一项扎实且有意义的贡献，更新了我们对 LLM 如何形成与运作的理解，但网络上的反应被大幅夸大，有人将其比作《Attention Is All You Need》，或声称机制可解释性已接近解决。其核心理论把智能设想为暴力枚举，神经元如同复合词，每个新概念都获得自己的神经元；但作者指出，论文淡化了残差流和注意力在神经元标签之外所进行的大量处理。结论是：这是真正值得兴奋的进展，但这并不意味着可解释性很快会被解决、对齐问题会被处理，或者我们拥有了针对 AI 的 MRI；枚举神经元仍只是整体谜题的一小部分，而机制可解释性的默认结局是古德哈特定律。
This article examines Nick Cammarata's claim that neural networks are already clean, meaning most neurons in larger LLMs represent a singular concept rather than overloading unrelated concepts through superposition. The author argues that while the work is a solid and meaningful contribution that updates our understanding of how LLMs form and operate, the online reaction has been substantially overstated, with some comparing it to Attention Is All You Need or claiming mechanistic interpretability is nearly solved. The core theory pictures intelligence as brute force, where neurons act like compound words and each new concept gets its own neuron, but the author notes the paper downplays how much processing occurs in the residual stream and attention outside neuron labels. The conclusion is that this is genuine progress worth getting excited about, but it does not mean interpretability will soon be solved, that alignment will be handled, or that we have an MRI for AI; enumerating neurons remains a small fraction of the overall puzzle, and the default outcome of mechanistic interpretability is Goodhart's Law.
Goodfire 的 Nick Cammarata 提出了一个大胆的主张：神经网络已经清晰。这是他在各种由此产生的在线讨论之后的后续发言。
Nick Cammarata of Goodfire makes a bold claim: Neural Networks are Already Clean. This was his follow-up after various resulting online discussions.
这显然是出色的工作，告诉了我们新的重要信息。问题是，这是一个多大的更新，它代表了多少进展，以及未来的前景如何？
This is clearly good work that tells us new and important things. The question is, how big an update is this, how much progress does it represent, and what is the outlook going forward?
这里的反应从超级乐观的观点——‘这意味着机制可解释性是可解决的’，甚至‘几乎已解决’——到与著名论文《Attention is All You Need》的比较，不一而足。
The reactions here range from the super optimistic takes of ‘this means mechanistic interpretability is solvable,’ or even ‘all but solved,’ with comparisons to the famous paper Attention is All You Need.
Dean W. Ball（OpenAI）：关于这一点有太多要说的，我将会说，但哇，祝贺 Nick 和 Goodfire。这可能是深度学习历史上最重要的单日之一，或许远不止于此。
Dean W. Ball (OpenAI): There is so much to say about this, and I will, but WOW, congratulations to Nick and Goodfire. This may be one of the most important single days in deep learning history, and perhaps much more than that.
无论如何，这是对理解的不懈追求的胜利。
No matter what, this is a victory for the dogged pursuit of understanding.
roon（OpenAI）：当 nick 告诉我他取得了一些可解释性突破时，呃，他其实还低估了这件事。我认为这可能会被铭记为科学史上的一个重大时刻。好消息好到连 Eliezer 都得下调他的 pdoom。我们可以开始把对齐变成一门“常规科学”了。
roon (OpenAI): when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even Eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'
这篇论文是扎实且有意义的贡献，但在某些方面被夸大了。可以兴奋，但也有些人需要冷静一下。我们实际上并没有得到从 1.7% 到 90% 的 LLM 神经元可解释性的同口径跃升。
This paper is a solid and meaningful contribution that is in some ways overstated. Get excited, but also there are those who need to take a chill pill. We did not actually get an apples-to-apples jump from 1.7% to 90% interpretability of LLM neurons.
我的解读是 Eliezer 略微下调了他的 p（doom），但幅度并不大。
I interpret Eliezer as slightly lowering his p(doom), but not all that much.
Evžen Wybitul：发生了什么？我时间线的一半是 OAI 的非可解释性研究者，把这篇文章描绘成基督再临；另一半是可解释性研究者（包括 Leo Gao），说它……还行。
Evžen Wybitul: What's happening? Half of my tl is non-interp people from OAI painting the post as the second coming of Christ and the other half is interp researchers (including Leo Gao) saying it's... fine
这篇文章是我试图理清这件事的尝试。目前我的立场大致在中间。这是一次关于 LLM 和心智如何形成与运作的重要更新。它告诉我们如何着手一些有用的任务，并降低了我们对某些任务难度的估计。很棒的工作，前提是这项工作经得起检验。整件事很有真实感，也很有一个人出于好奇心的感觉。你乐于看到这一点。
This post is my attempt to sort this out. Currently I am landing somewhere in the middle. It is an important update about how LLMs and minds form and operate. It tells us how to go about some useful tasks and reduces our estimate of some task difficulties. Great work, assuming the work checks out. The whole thing very much has a ring of truth, and of a person being curious. You love to see it.
它并未告诉我们可解释性很快就会被解决，或即将成为一门科学，更不用说对齐很快就能得到处理。在某些地方，炒作已经相当失控。
What it does not tell us is that interpretability will soon be solved, or is about to be made into a science, let alone that alignment will soon be handled. The hype got rather out of hand in some places.
8. 神经元有时会共享清晰的主题。
8. The Neurons Sometimes Share Clear Themes.
13. 但我们可以打开盒子，蟾蜍说。
13. But We Can Open The Box, Said Toad.
16. 这就是全部，再无其他。
16. That's All There Is, There Ain't No More.
17. 机制可解释性的默认结果是古德哈特定律。
17. The Default Outcome of Mech Interp is Goodhart's Law.
在我讨论各方反应或选边站队之前，先列出我的要点。至少读这么多原始材料是值得的。通读全文也是一种选择，但篇幅很长，行文容易让人读不下去，而且内容可能重复。
Before I cover reactions or take a side, here are my highlights. It is worth reading at least this much of the primary source. Reading the whole thing is an option, but it is long and the writing can be easy to bounce off of and can repeat itself.
Nick Cammarata，Goodfire：早期模型生物中多义性（polysemanticity）的成因并非叠加（superposition），而是若干普通且大多可修复的问题。
Nick Cammarata, Goodfire: The cause of polysemanticity in early model organisms wasn't superposition, but several ordinary, mostly fixable issues.
我们现在能够理解参数量高达数万亿的模型中的大多数神经元，并认为机制可解释性（mechanistic interpretability）的前景可能远比我们想象的更好。
We can now understand most neurons in models up to trillions of parameters, and think the outlook for mechanistic interpretability may be far better than we thought.
我认为，关于神经网络的本质，我有一个好消息。
I think I have good news about the nature of neural networks.
在花费数月研究图像模型和语言模型中的神经元与回路之后，我开始相信，网络自身的机制就是其清晰可读的构建单元，而机制可解释性的前景可能远比我此前担心的要好。
After many months studying neurons and circuits in image and language models, I've come to believe a network's own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I'd feared.
多义性与叠加的经典证据曾让该领域大多数人（包括我）相信网络会任意地打包概念，但这些证据被误读了。
The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks pack concepts arbitrarily was misread.
看似任意的打包，其实大多是一种我称之为递归泛化的模式，其令人担忧的程度要低得多。
What looked like arbitrary packing was mostly a motif I'll call recursive generalization, which is much less concerning.
核心主张是：对于更大的 LLM，大多数神经元表示一个单一概念以及围绕该概念聚集的事物，或许还带有小的分支，并会用到一些低层激活。
The central claim is that for larger LLMs, most neurons represent a singular concept and things clustered around that concept, perhaps with small offshoots and some use of low-level activations.
……现代语言模型的可解释性也远高于人们通常所认为的。
… Modern language models are also far more interpretable than generally believed.
……借助更好的工具和现代模型，情况看起来大不相同。在对 Qwen3-32B 的初步审计中，我们估计大约 90% 的神经元极点只有一个主要职责，通常带有小的相关分支；另有约 7% 有两个相关职责；只有约 2–4% 尚未被理解。
… With better tools and modern models, the situation looks quite different. In preliminary audits of Qwen3-32B, we estimate that roughly 90% of neuron poles have one main job, often with small related offshoots; another ~7% have two related jobs; and only ~2-4% are not yet understood.
稠密模型和混合专家（Mixture of Experts）模型似乎大致相似。
Dense and Mixture of Experts models seem roughly similar.
我们在参数量高达数万亿的模型（如 Kimi K3）中发现了类似结果，配套论文将对此进行更详细的介绍。
We found similar results in models up to trillions of parameters, like Kimi K3, which the companion paper will cover in more detail.
以前的工作理论认为，模型通过叠加将不相关的概念过载到神经元上，而尼克的理论是，你没有足够努力地思考以识别出统一的概念。每个神经元都是一个泛化，大多数是我们只要观察就能轻易理解的泛化。
Where previously the working theory was that models overloaded neurons with unrelated concepts in superposition, Nick's theory is you did not think sufficiently hard to identify the unifying concept. Each neuron is a generalization, mostly ones we can readily understand if we look.
……更重要的是为什么事物首先会共享一个神经元。大多数剩余的明显多义性似乎是一种我称之为递归泛化的模式。这很简单：随着神经元的生长，它会像树枝长出子分支一样长出分支。例如，一个检测“iy”的神经元发展到检测“diy”，然后检测“do it yourself”，而后者根本不包含“iy”。
… What matters a lot more is why things share a neuron in the first place. Most of the remaining apparent polysemanticity seems to be a motif I call recursive generalization. It's simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected "iy" grew to detect "diy," and then "do it yourself," which doesn't contain "iy" at all.
……因此，对多义性有两种解释，而我帮助提出了这两种解释。在某些方面，它们是对立的：泛化将相似的事物分组在一起，而叠加将不相关的事物分组在一起。叠加是该领域押注的解释，直到最近我也是如此。但我现在认为递归泛化基本上是正确的。
… So there were two explanations for polysemanticity, and I helped propose both. In some ways, they're opposites: generalization groups similar things together and superposition groups unrelated things together. Superposition is the one the field bet on and, until recently, so did I. But I now think recursive generalization was basically right.
……可解释性有着惊人的势头：就像拼图游戏或填字游戏一样，你获得的任何理解都会开始约束系统的其余部分。
… Interpretability has a surprising amount of momentum: like a jigsaw puzzle or crossword, any understanding you gain starts constraining the rest of the system.
……以一个神经元为例，它追踪的是‘当廉价购买的结果比昂贵替代品更好时的一种正义感’。追踪这种感觉或社会线索的神经元是我们发现的最常见的神经元之一。
… Take a neuron that tracks 'a sense of righteousness when a bargain purchase turns out to work better than the expensive alternative'. Neurons that track a feeling or social cue like this are some of the most common we've found.
这把智能描绘成一种蛮力。神经元就像德语中的复合词。
This pictures intelligence as brute force. Neurons are compound words in German.
新概念？再去拿一个神经元。然而，它确实有效。
New concept? Go get another neuron. And yet, it works.
……现代神经网络坏消息是它们极其庞大。好消息是，这种庞大有帮助：随着图谱逐渐填满，每一种解释都被周围的一切紧密约束，而更大的模型会学到更窄的神经元，这些神经元从一开始就更容易被识别。
… The bad news about modern neural networks is that they're enormous. The good news is that the enormity helps: as the map fills in, every explanation is tightly constrained by everything around it, and bigger models learn narrower neurons that are easier to recognize in the first place.
从这个意义上说，理解整个网络可能比只理解我们最关心的那些部分更容易。
In this sense, understanding the whole network may be easier than understanding just the parts we care about most.
Nick 认为，这意味着先解决整个谜题、理解一切，然后再对你感兴趣的特定区域进行测量，要比试图直接从一堆神秘神经元中拽出一个像“欺骗”这样的模糊概念更容易。
Nick argues that this means that it is easier to solve the full puzzle and understand everything, and then take a measurement of your specific area of interest, than to try to directly pull a fuzzy concept like 'deception' out of a pool of mystery neurons.
……“欺骗”是一个有用的初始框架，但促成模型欺骗行为的那些构件很可能具有与对齐相关的微观结构，若不深入进去观察，就很难了解。
… 'deception' is a useful initial frame, but the building blocks that contribute to deceptive behavior by models probably have alignment-relevant microstructure that will be hard to know without going in and looking.
……所有这些让我回到一个旧的希望：直接理解神经网络，研究其机制以及机制之间流动的数据，而不先去修复或理清它。综合起来，这个论证有四步。
… All of this has brought me back to an old hope: understanding a neural network directly, by studying its mechanisms and the data flowing between them, without first trying to fix or untangle it. Put together, the argument has four steps.
1. 网络只使用一种学习原理，而非两种（即便并非处处如此，也几乎足够普遍，以至于对研究而言无关紧要），因此开始对它们进行常规科学研究并不存在障碍。
1. Networks use one learning principle, not two (if not everywhere, then nearly enough that it doesn't matter for study), so there's no blocker to beginning ordinary science on them.
2. 借助现代模型与工具，这项科学研究的进展远超大多数人的预期。
2. With modern models and tools, that science goes much better than most people expect.
3. 理解具有累积效应，因此随着图谱逐渐填满，它会变得越来越容易。那么，至少网络自身的机制应当成为工具箱的一部分。
3. Understanding compounds, so it gets easier as the map fills in. At minimum, then, a network's own mechanisms should be part of the toolkit.
4. 高度推测性：这可能正是我们所需要的大部分内容，而可解释性或许远比我们设想的更易处理，甚至可能在近期实现。
4. Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we've assumed.
这正是深度学习整体的模式：在荒诞规模下的简单方法胜过精巧的方法。
That's been the pattern for deep learning in general: the simple thing at absurd scale beats the clever one.
这一主张极为大胆且激进，必将直接受到质疑。多数人似乎已从这篇论文中更新了认识：比我们此前所想的更多神经元背后存在中心化的单一概念。Nick 试图更进一步，主张这就是普遍发生的情况。
This claim is maximal and bold, and will get directly challenged. Most people seem to have updated from this paper that more neurons have central singular concepts behind them than we previously thought. Nick is trying to go further, and say that this is what is going on universally.
……我打开那张早已看过无数次的猫与汽车图像网格，做了一件此前从未做过的事：我花了几秒钟琢磨，这会不会其实是同一个东西。几乎瞬间就清楚了：这不是猫与汽车神经元，而是一个 V 形神经元。
… I opened up the cat and car image grid I'd seen hundreds of times before and did something I'd never done: I spent a few seconds wondering what one thing this might be. Almost instantly it was obvious: it's not a cat and car neuron, it's a V-shape neuron.
未来还有 Lady Reds：这个极其神秘的神经元究竟是什么，竟让机制可解释性（mechinterp）研究在过去三年里走进死胡同？
there are Lady Reds in the future: What is this extremely mysterious neuron that set mechinterp going on a dead end for the last 3 years?
它就是字面意义上的
It's literally
下次当你觉得自己永远也弄不明白某件事，或者认定别人肯定早就问过那些基本问题、考虑过那些显而易见之处时，请记住这一点。也许他们确实问过、考虑过。也许并没有。
Remember this the next time you think you can never figure anything out, or that surely people have already asked the basic questions and considered the obvious. Maybe they have. Maybe they haven't.
论文（续）：……InceptionV1 拥有一百多个这类泛化的姿态神经元，每一个都是针对某种姿态的干净检测器，而这种姿态为汽车、狗以及其他物体所共有；据我所知，此前无人注意到它们。
Paper (resuming): … InceptionV1 has more than a hundred of these generalized pose neurons, each one its own clean detector of a pose shared across cars, dogs and other objects, and to my knowledge no one had noticed them.
……我常常可以通过这样提问来解开对某个神经元的理解障碍：如果完全不能使用名词词组，我会如何向别人描述这个神经元？
… I can often unblock my understanding of a neuron by asking: if I couldn't use groups of nouns at all, how would I describe this neuron to someone else?
……我们应当把一个表示两三种含义的神经元视为一个带有若干分组含义的神经元。这种分组对网络来说大概并不构成问题：这些含义之所以聚在一起，是因为从网络的角度看它们是相似的。
… We should treat a neuron that means two or three things as one neuron with a few grouped meanings. The grouping probably wasn't a problem for the network: the meanings sit together because, from the network's point of view, they're similar.
……多义性（polysemanticity）往往更多是关于观察者的论断，而非关于神经元的论断。它说明目前还没有好的解释。造成这种情况可能有几个原因：
… Polysemanticity is often more a claim about the observer than the neuron. It says there's no good explanation yet. There's a few reasons that could be:
1. 工具不佳：我们的透镜或显微镜不够好
1. Bad tools: our lens or microscope isn't good enough
2. 解释不佳：由这些工具产生的解释
2. Bad explanations: created from those tools
3. 不良学习：由于糟糕的架构、训练数据或其他原因，网络学习效果很差
3. Bad learning: the network learned poorly due to bad architecture, training data, or something else
4. 递归泛化：在泛化过程中，该神经元分叉出若干相关的方向
4. Recursive generalization: while generalizing, the neuron branched off in a few related directions
5. 叠加：一种学习原理上的变化，即网络将不相关的事物打包进同一批神经元，因为它们很少发生冲突。
5. Superposition: a change in learning principle where the network packs unrelated things into the same neurons because they rarely collide.
……在一个前沿语言模型中，我发现了一个神经元，它主要对“pr”激活，但也会对一堆 Base64 URL 字符串以及一些孟加拉语激活。我以为这终于是一个任意混合的例子。但解码这些 URL：是“pr”词。翻译孟加拉语：是“pr”。
… In a frontier language model, I found a neuron that mostly fired on "pr", but also on a bunch of Base64 URL strings as well as some Bengali. I thought this must finally be an arbitrary mixture. But decode the URLs: "pr" words. Translate the Bengali: "pr".
……在一个稠密 Qwen 模型中，一位同事发现了一个神经元，标准的 top-k 自动化可解释性方法将其列为“多义逗号神经元”。但更仔细地观察后发现，它实际上追踪的是科罗拉多：电话号码、俚语，以及与其他科罗拉多神经元的连接。
… In a dense Qwen model, a coworker came across a neuron that standard top-k automated interpretability lists as a "polysemantic comma neuron". But looking more closely, it turned out to track Colorado: phone numbers, slang, and connections to other Colorado neurons.
对它进行引导（steering）会让模型想要去徒步。逗号是它存储科罗拉多上下文迄今运行摘要的位置。总体而言，注意力机制似乎会聚集在标点符号上，我们称之为摘要母题（summary motif）。一旦你了解了这一点，许多类似的神经元就更容易理解了。
Steering it made the model want to go hiking. The commas were where it stored a running summary of the Colorado context so far. In general, attention seems to collect on the punctuation, which we call the summary motif. Once you know it, a lot of similar neurons get easier to understand.
……在另一个案例中，一个神经元看起来像一团随意的乱麻，没有任何强烈的激活。事实证明，它在健康类 SEO 垃圾网页中寻找特定迹象，而一旦我们拥有足够广泛的数据集，它就可以被理解了。
… In another case, a neuron looked like an arbitrary mess, with nothing firing strongly. It turns out it looked for specific signs within pages of health SEO spam, and once we had a sufficiently broad dataset, it was understandable.
Nick 确实注意到了那种彻底陷入“这不是巧合，因为从来就没有巧合”式思维所导致的骷髅头。
Nick does notice the skulls that result from going full ‘This Is Not a Coincidence Because Nothing Is Ever a Coincidence.’
……这有犯相反错误的风险，即对空白处的解释：假设每个神经元都必须有一个合理的故事。有两件事让它不那么令人担忧。第一，其他每个原因都可以被检验。……第二，真正的解释往往范围狭窄且易于检验。一旦你怀疑某个神经元是科罗拉多神经元，就引导它，看看它会说什么。
… This risks the opposite error, an explanation of the gaps: assuming every neuron must have a sensible story. Two things make it less worrying. First, each of the other causes can be tested. … Second, real explanations tend to be narrow and easily testable. Once you suspect a neuron is a Colorado neuron, steer it and see what it says.
强假设是，所有神经元都会拥有像科罗拉多这样的核心概念。更强的假设是，不会同时还有无关的额外概念在幕后发生。
The strong hypothesis is that neurons will all have central concepts like Colorado. The even stronger hypothesis is that there will not also be unrelated additional concepts happening behind the scenes.
……我过去把神经元想象成人类形态的名词：一个是汽车，另一个是曲线。现在，我更倾向于把它们看作控制流。神经元与其说关乎它看到了什么，不如说关乎它对流入的信息流做了什么，而且它的职责往往是对网络的某个区域进行门控。
… I used to think of neurons as human-shaped nouns: one is a car, another is a curve. Now I think of them more as control flow. A neuron is less about what it sees than about what it does to the flow coming in, and often its job is to gate a region of the network.
……对于理解神经元而言，其负向侧的重要性几乎不亚于正向侧，即使在负向侧被截断的 ReLU 网络中也是如此。
… The negative side of neurons matters almost as much as the positive side for understanding them, even in ReLU networks where the negative side is cut off.
……有些神经元在很低的激活范围内仍能保持其含义；另一些则急剧衰减，就像上面提到的那个“圣诞神经元”在其最大值约 5% 处便断崖式下降。有些神经元在一本书中只激活几次，另一些则在一页中激活几次。
… Some neurons hold their meaning far down the range; others drop off a cliff, like the Christmas neuron above at about 5% of its max. Some fire a few times a book, others a few times a page.
核心理论是：也许智能是一个比我们想象中更简单的概念？它不过是一系列相互关联、彼此链接的连贯概念？是否存在一个“《银河系漫游指南》状态单元”，它在标题和“答案”槽位上达到峰值，并对所有 42 都微弱激活？是否存在一个专门对应“聚变永远在固定时间之后”这个笑话的神经元，它会在其他类似的重复承诺上激活？
The core theory is that perhaps intelligence is a simpler concept than we think? It is all a series of coherent concepts with links? There's a 'Hitchhiker's Guide state unit that peaks on the title and the Answer slot' that weakly fires on all 42s? There is a neuron specifically for the joke about fusion always being a fixed amount into the future that fires on other similar repeated promises?
我们看到的是，大约 90% 的神经元基本上只做一件事，其递归泛化的分支通常也与之相关。……我从未见过一个神经元会做十件互不相关的事。
What we see is about 90% of neurons having basically one job, with recursive generalization offshoots that are usually related. … I've never seen a neuron with something like ten unrelated things it does.
Nick Cammarata，Goodfire：这是一个仅使用神经元构建的 Kimi K3 归因图。据我们所知，这是首个公开发表的、针对数万亿参数模型的归因图。当被问及捡到钱包该怎么办时，它把钱包、现金和失主身份证件与失物招领和诚实联系起来，并回答“归还”。我们尚未开始这种规模的全面审计，但希望很快能够开展；不过到目前为止，大多数内容在直觉上都是可解释的。
Nick Cammarata, Goodfire: Here's an attribution graph from Kimi K3 using only neurons. As far as we know, it's the first published attribution graph of a multiple trillion parameter model. Asked what to do with a wallet it found, it links the wallet, the cash and the owner's ID to lost property and honesty, and answers "return". We haven't begun full audits of this scale yet, and hope to soon, but so far most things have been intuitively interpretable.
把简单的事情做好。回响。感觉。没有论文或文章，只有对论文和文章的引用。一个彻底的苦涩教训。越苦涩，教训就越简单。
Simple things done well. Echoes. Vibes. No papers or essays, only references to papers and essays. A fully bitter lesson. The more bitter, the easier the lesson.
2023 年，在第一个自动化可解释性项目期间
In 2023 during the time of the first automated interpretability project
我原以为小型简化模型会是可解释性研究的最佳起点，因为需要考虑的细枝末节更少，而且规模小意味着需要研究的东西也更少。但现在我认为情况可能恰恰相反。
I thought small simplified models would be the best starting place for interpretability, since there are fewer bells and whistles to think about, and small size means fewer things to study. But now I think it may be the opposite.
然后是残差流和注意力机制。这篇文章似乎试图掩盖一个事实：许多正在发生的事情其实发生在神经元标签之外。
Then there’s the residual stream and attention. The essay feels like it’s trying to hide how much of what is going on is happening outside of the neuron labels.
对残差流的担忧并不是我们关心的事物比维度多。这是一种连续版本的打包：流形相互交织，接受一些干扰以换取更多存储空间。完全排除这种可能性或许不可能。但在实践中，它并未成为瓶颈。
The worry with the residual stream isn’t that there are more things we care about than dimensions. It’s a continuous version of packing: manifolds woven through each other, accepting some interference in exchange for room to store more. Ruling that out entirely may not be possible. But in practice it hasn’t been a bottleneck.
我们可以读取神经元写入残差流的内容以及它们从残差流中读取的内容，归因图清晰呈现，并且我们没有看到残差流积累干扰后又将其消除。
We can read what neurons write to the residual stream and what they read from it, attribution graphs come out clean, and we don’t see the stream accumulating interference only to undo it later.
……一个流仅仅通过将事物放置在正确的方向上，就能容纳远超其维度的信息，所以这本身并不是打包。它主要在你想要模型所有特征的大列表时才重要，而人们常常正是为此而求助于残差流。
… A stream can hold far more than it has dimensions just by placing things in the right directions, so this alone isn’t packing. It mostly matters if you want a big list of all the model’s features, and people often go to the stream for exactly that.
……我们对语言模型的残差流研究较少，但所见符合相同的规律。对于理解更高层次的结构和模式，残差流中尚未有任何东西阻碍我们。
… We’ve looked at language models’ residual streams less, but what we’ve seen fits the same story. For understanding higher-level structure and motifs, nothing in the residual stream has blocked us so far.
……我们在注意力机制上的进展不如在神经元上的进展，而且这大多超出了本文的讨论范围。说实话，我不确定是否有人对注意力机制的理解足够深入，以至于能自信地断言：不可读性就是我们关心的那种意义上的叠加。
… We've made less progress on attention than on neurons, and it's mostly outside the scope of this essay. To be honest, I'm not sure anyone understands attention well enough to confidently say that illegibility is a superposition in the sense we care about.
……总体而言，我们认为注意力机制和神经元并非如此可分离。理解注意力机制把信息放在哪里，对于理解随后处理这些信息的神经元非常有帮助。
… Overall, we don't think attention and neurons are so separable. Understanding where attention is putting information is quite helpful for understanding the neurons that then work with that information.
这是 Nick 似乎越界的一个地方。是的，我们可以在不知道是什么机制分配你的注意力的情况下，观察注意力把你引向何处，然后再观察被触发的神经元。我不认为这能让你“摆脱”理解那个过程的需要。无论如何，观察神经元是值得的。但这并不能让你预测未来 LLM 的输出，也不能让你完全理解它是如何思考的。
This is one place Nick seems to overreach. Yes, we can look at where attention guides you without knowing what mechanism allocates your attention, and then look at the neurons that get triggered. I don't think this 'gets you out of' having to understand that process. It is worth looking at the neurons anyway. This does not allow you to predict future LLM outputs or fully understand how it thinks.
这是否意味着，如果你没有取得更多进展，你就可以“坐镇于重要的神经元之上”来防范不轨行为，或者以其他方式做好防御？在某种程度上可以，但我会假设，一个知道这类监控的 LLM 能够绕过它，并想出如何“不去想”你所寻找的东西。
Does this mean you can 'sit on the neurons that matter' to guard against shenanigans or otherwise play good defense, if you don't make more progress? To some extent, but I would assume that an LLM that knows about this type of monitoring would be able to work around this, and figure out how 'not to think about' what you look for.
关于所有神经元为何如此简单的理论是：这很简单。
The theory of why all the neurons is simple: that this is simple.
……我认为事物最终变得如此有条理的部分原因是，以那种方式构建起来更快。一旦神经网络中有了一个包含汽车部件的汽车区域，在那里添加边际汽车比在其他地方构建第二个汽车区域更容易。我认为这与电路竞赛和梯度饥饿的想法密切相关。
… I think the reason things end up so organized is partially because it was faster to build that way. Once you have a car area of the neural network with car parts, it’s easier to add the marginal car there than to build a second car area somewhere else. I think this is closely related to ideas of circuit racing and gradient starvation.
……即使地图是可理解的，那也不意味着你已经理解了它。我们仍然必须进行猜想与反驳的普通科学：猜测某物是什么，尝试打破猜测，并保留幸存下来的部分。网络是奇怪的，但它的奇怪性会重复，因此一个经得起审视的猜测往往会多次带来回报。
… Even if the map is understandable, that doesn’t mean you’ve understood it. We still have to do the ordinary science of conjecture and criticism: guess what something is, try to break the guess, and keep what survives. The network is strange, but its strangeness repeats, so a guess that survives scrutiny often pays off many times over.
……解决可解释性可能与实现 AGI 有共同之处：目标会随着你的接近而移动。总会有另一个微弱的激活可以指出。
… Solving interpretability may share something with achieving AGI: the goalposts move as you get closer. There’s always another weak activation to point to.
……对神经元中小误差的怀疑背后，是一种恐惧：安全性将归结为我们遗漏的某个激活，即欺骗隐藏的地方。我担心的风险不同：就像身处一个组织良好的城市却不知道如何思考它，因为我们不是它的原住民。
… Behind doubts about small errors in neurons is a fear that safety will come down to some activation we missed, the place where deception hides. The risk I worry about is different: like being in a well-organized city and not knowing how to think about it, because we aren’t native to it.
……我认为入门教科书中的简单叙述基本上是正确的：神经元会泛化、协同工作，并在各层之上构建抽象。在这一过程中，我们曾把这一图景过度复杂化，但更深入的研究又将其还原。如果网络本身并不是一个结，那么把它解开反而可能让它看起来像一个结。
… I think the simple story from introductory textbooks is basically right: neurons generalize, work together, and build abstractions over layers. Somewhere along the way we overcomplicated that picture, but looking harder brought it back. If the network isn't a knot, untangling it can make it look like one.
Sauers 注意到 Qwen3-8B 中不同特征之间存在神经元重叠的模式。这与 Nick 的理论似乎高度一致。
Sauers notices patterns of overlap of neurons between different features in Qwen3-8B. This seems highly consistent with Nick's theory.
roon：我想说，即便是在一场真正的科学革命中，你也不会指望该领域此前的专家最先被说服。事实上，他们可能是最后被说服的。
roon: I will say that even in a bona fide scientific revolution, you wouldn't expect the field's previous experts to be the first convinced. In fact, they may be the last.
Lake Snyder：一个有趣的二分法是，Roon 完全被说服了，而（稍微）年长一些的守旧派却持怀疑态度。
Lake Snyder: It's an interesting dichotomy that Roon is completely convinced but the (slightly) older guard is skeptical.
roon：老实说，如果他说的是真的，那么在这篇论文之前就身处机制可解释性（mechinterp）领域，可能本身就是一种信息危害。
roon: I honestly think that being in mechinterp pre this paper may have been an infohazard if what he is saying is true.
roon：我觉得我对神经网络是什么的理解，仿佛被向前推进了五年左右。它让我重新找回了研究深度学习之谜时那种纯粹的快乐。神经元已经是干净的了！它们构成了一种奇妙的异质本体论！
roon: I feel like my understanding of what a neural net is has been catapulted into the future by five years or something. It has brought back a lot of the sheer joy I had for studying the mystery of deep learning. Neurons are already clean! They form a wonderful alien ontology!
我们在开头已经看到了一些炒作。这里还有更多炒作。
We saw some hype up top. Here is some more hype.
Andrew Curran：对我来说，这将是未来的一次重大积极更新。我早就放弃了在超级智能到来之前实现任何真正水平的可解释性。我停止写这个话题，因为那太令人沮丧了。很难夸大这可能有多重要。
Andrew Curran: This would be a massive positive update for me on the future. I had long since given up on reaching any real level of interpretability before superintelligence arrived. I stopped writing about it because of how demoralizing it was. Hard to overstate how big this may be.
……为了帮助人们理解这看起来有多遥远，这是去年四月的 Dario——在当时，这是关于该主题的一个巨大的积极更新——说我们将在 5-10 年内实现它。现在在许多时间线上，包括我自己的，这都太晚了。
… To help people understand how far away this seemed, this is Dario in April of last year - in what at the time was a huge positive update on this subject - saying we would reach it within 5-10 years. On many timelines now, including my own, this would have been too late.
Dario Amodei（2025 年 4 月）：一方面，最近的进展——尤其是关于电路和基于可解释性的模型测试的结果——让我觉得我们即将大规模突破可解释性。尽管我们面前的任务艰巨无比，但我能看到一条现实的路径，使可解释性成为一种复杂而可靠的方法，用于诊断即使是非常先进的 AI 中的问题——一个真正的“AI 的 MRI”。事实上，按照目前的轨迹，我强烈押注可解释性将在 5-10 年内达到这一点。
Dario Amodei (April 2025): On one hand, recent progress—especially the results on circuits and on interpretability-based testing of models—has made me feel that we are on the verge of cracking interpretability in a big way. Although the task ahead of us is Herculean, I can see a realistic path towards interpretability being a sophisticated and reliable way to diagnose problems in even very advanced AI—a true “MRI for AI”. In fact, on its current trajectory I would bet strongly in favor of interpretability reaching this point within 5-10 years.
这正是 Eliezer Yudkowsky 将在本文后面咆哮的错误。是的，这是进步。不，这不是“AI 的 MRI”。那还很遥远。
This is exactly the mistake Eliezer Yudkowsky will be ranting about later in this post. Yes, this is progress. No, this is not an ‘MRI for AI.’ That is still a long way off.
Eric Ho（Goodfire）：过去一年里，看着 @nickcammarata 把这件事做出来，是一种巨大的荣幸和快乐。一周又一周，一个实验又一个实验，他解开了我们对神经网络思考的混乱纠缠，揭示了真相。
Eric Ho (Goodfire): it’s been an enormous privilege and joy to watch @nickcammarata cook this up over the last year. week after week, experiment after experiment, he unraveled the tangled mess of our thinking about neural nets to reveal the truth.
在我们每周的散步中，我从来不知道会期待什么。每次对话都带来大量关于 AI 本质的新理论，一些猜想摇摇欲坠于疯狂的边缘，而另一些似乎用一句话就穿透了多年的困惑。
I never knew quite what to expect on our weekly walks. Each conversation brought a torrent of new theories about the nature of AI, some conjectures teetering on the brink of madness, and others that seemed to cut through years of confusion with a single sentence.
Nick 是我们时代伟大的科学家之一。他的直觉和研究品味超凡脱俗，但让他与众不同的是他无与伦比的能力：捕捉事物真正本质，而不迷失在细节中。如果你对 AI 好奇，我强烈推荐你读他的整篇文章。它很长，但写得很美。
Nick is one of the great scientists of our time. His intuition and research taste are out of this world, but what sets him apart is his unmatched ability to capture the essence of what something truly is, without being lost in the details. I would highly recommend you read his entire essay if you're curious about AI. It's long, but written beautifully.
Nick 的研究让我对 AI 对齐的未来更加乐观。我希望它也能激励你。
Nick's research makes me far more optimistic about the future of AI alignment. I hope that it can inspire the same for you.
Goodfire：多年来，神经网络看起来难以捉摸，就像需要解开的乱结。我们不再认为情况如此。AI 模型比我们想象的更具可解释性。我们认为这对可解释性来说是个好消息。
Goodfire: For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.
Goodfire 的@nickcammarata 共同撰写了《Zoom In》，这篇论文引入了多义性和叠加。他花了几个月时间重新审视可解释性发展所基于的视觉模型生物，并研究高达数万亿参数的现代语言模型。他的新文章解释了早期领域（包括他自己）错在哪里，为什么今天有理由乐观，以及为什么可解释性可能有一个苦涩的教训。
Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.
即便你认为这是一件大事，部分炒作也已经失控了。
Even if you think this is a big deal, some of the hype got crazy.
Adam Majmudar（OpenAI）：是的，这是本年度最重要的成果，其重要性堪比《Attention Is All You Need》。它把我们从此前“对齐是否可解”的疑问，带到了“我们或许能在明年解决对齐”的阶段。并且最终解锁了由 RSI 所赋能的真正递归自我对齐。
Adam Majmudar (OpenAI): Yea this is the most important result of the year, at the level of significance of "Attention Is All You Need". It brings us from "is alignment even solvable" to "we may solve alignment in the next year." And finally unlocks true recursive self-alignment enabled by RSI.
1/ 神经网络从根本上并非人类无法理解的
1/ neural networks are not fundamentally unintelligible to humans
2/ 神经网络从根本上就是干净的
2/ neural networks are fundamentally clean
3/ 人类的语言与神经网络的语言是同一种语言
3/ the language of humans and the language of neural networks is the same
4/ 对齐可以像 RSI 一样快速发展。递归自我改进解锁了递归自我对齐以及对齐的缩放定律。
4/ Alignment can progress as fast as RSI. Recursive self-improvement unlocks recursive self-alignment and a scaling law for alignment.
adammaj：我绝对不认为这是支持 RSI 的论据，也不认为对齐问题已经解决。这只是在原本黯淡的进展速度前景中一个乐观的理由，这正是我想表达的。
adammaj: I definitely don’t think this is an argument for RSI or that alignment is solved. It’s just a reason for optimism in an otherwise bleak landscape of the rate of progress, which is what I intended to express.
adammaj：在阅读了一些评论后，让我补充一些非常重要的细微差别：
adammaj: Let me add some VERY important nuance after reading some comments:
当我说“我们可能在未来一年内解决对齐问题”时，我应该说的是“我们可能解决可控性”——我并不是指整个对齐问题。价值观是一个完全不同的问题。而且一年是雄心勃勃的。也许 2-4 年更现实。但你必须考虑到进展速度的快速增长，这不断超越人类的预期。
When I say “we may solve alignment in the next year” I should have said “we may solve controllability” — I don’t mean the whole alignment problem. Values is a completely different problem. Also 1 year is ambitious. Maybe 2-4 years is more realistic. But you have to price in the rapid increases in rate of progress which keeps beating human expectations.
将其与“注意力机制就是你所需要的一切”相比较是一个相当激进的比较，我指的不是今天，而是极限情况（即，如果 Nick 在这里所说的一切都是真的，并且研究方向继续如预期发展——让我们看看他们何时发布技术论文）。现在下结论可能相当愚蠢，应该更多地将其视为关于这个方向可能产生的影响的预测。
Comparing it to “attention is all you need” is a pretty aggressive comparison, and I mean it not today but in the limit (i.e., if everything Nick says here is true and the research direction continues to play out as expected — let’s see when they release their technical paper). Probably pretty stupid to say conclusively now, should be read more like a prediction about the impact this direction could have.
从这里解决可解释性问题仍然会很困难，而且是一个非常庞大/复杂的问题。我个人不是可解释性专家，但评论中有优秀的可解释性研究人员补充了细微差别。但这似乎是一个非常重大的更新，表明这种方法比预期更有前景，并且可能会成功。
Solving interpretability from here will still be hard and is a very large/complex problem. I am no interpretability expert personally, but there are good interpretability people in the comments adding nuance. But this seems like a very big update that this method is much more promising than expected and might work out.
Siddarth：抱歉，搞什么鬼，我感觉自己要疯了。这是一篇很酷的博客和令人兴奋的可解释性内容（干得好，Nick！），但我认为非常可疑的是，我们有一群 OpenAI 的人把它吹捧为这十年最伟大的 AI 进步。我想我知道为什么会这样，但我不想对时间线做太多推测。但总的来说，请大家负责任地更新你们的先验——对齐（甚至可解释性）远未解决。这不是明天就加速 RSI 的 OK 信号。
Siddarth: I'm sorry, wtf, I feel like I'm going insane. This is a cool blog and exciting interpretability stuff (good job Nick!), but I think it's very suspicious we have a bunch of OpenAI people hyping it up as the greatest AI advancement this decade. I think I know why this is happening but I don't want to speculate on the timeline too much. But in general, please update your priors responsibly, people -- alignment (and even interpretability) is FAR from solved. This is not the OK signal to accelerate RSI tomorrow.
我也感到震惊，竟然有人会将其解读为‘对齐可以像 RSI 一样快速进展’级别的价值。许多人，尤其是 OpenAI 的人，表现出惊人的急切，将其夸大到了 11 分，仿佛这能让我们假装一切安好。即使后来我们确实看到了一些回调，哇。
I too find it alarming that anyone could interpret this as 'alignment can progress as fast as RSI' levels of valuable. There was a remarkable eagerness of many, especially at OpenAI, to take it to 11 with how much this lets us pretend everything is fine. Even with the pullbacks we did see afterwards, wow.
我认为这主要是人们渴望一个希望的来源。生活中你需要这些：
I think mostly this is people desperate for a source of hope. You need those in life:
Dean W. Ball：我的儿子今天迈出了他独立的第一步。加上可解释性的东西，我真的充满了希望，这是我一段时间以来从未有过的。
Dean W. Ball: My son took his first independent steps today. Between that and the interpretability stuff I'm really overwhelmed with hope, which I have not been in some time.
你还需要一个故事，让你在认为一切都会成功的同时继续前进。这一切模式非常接近于继续循环：‘我修好了，我们现在可以回去训练模型了’，直到你有其他东西需要修补并假装没问题。
You also need a story that lets you keep advancing while thinking it is all going to work out. This all pattern matches very closely to continuing the cycle of ‘there I fixed it, we can get back to training the models now’ until you have something else to patch over and pretend is fine.
很难不看到这样的反应和梗图，并感受到其潜在论点（链接中并未明确声称）：‘那些傻瓜认为我们注定要失败，对齐因为叠加而无法解决，但现在我们知道叠加更少了，那些人是错的，这意味着我们将解决对齐。’不。
It is hard not to look at reactions and memes like this, and get the feel of the underlying argument (not explicitly claimed at the link) of ‘those fools thought we were doomed and alignment was unsolvable because superposition, but now that we know there is less superposition, those people are wrong, which means we will solve alignment.’ No.
起初，炒作给许多人留下了与 OpenAI 的 Drew Pusateri 相同的印象，即所有 AI 人士都超级兴奋，而媒体再次忽视了一个巨大的故事，Kelsey Piper 不得不指出媒体报道需要时间。从那时起，我认为很多人已经安定下来，采取了更温和的立场。
At first, the hype gave many the same impression as OpenAI’s Drew Pusateri, that all the AI people were super excited and the media was once again ignoring a huge story, and Kelsey Piper had to point out that media coverage takes time. Since then, I think a lot of people have settled into a more modest position.
我认为 Joshua 在某种程度上低估了这篇论文在改变我们部分理解和选项空间方面的程度，但即便是他也高估了这对于平凡安全技术的可靠性的暗示程度，同时也低估了世界施加优化压力的倾向。
I think Joshua is modestly underselling this paper in terms of how much it changes parts of our understanding and option space, but even he is overselling how much this implies about reliability of prosaic safety techniques, and also is underselling the propensity for the world to apply optimization pressure.
Joshua Achiam：我认为这很棒，非常酷！我很兴奋看到这里的见解转化为具有可量化成功率的工具，帮助我们识别潜在危害或引导神经网络远离风险。
Joshua Achiam: I think this is great and very cool! I am excited to see the insights here translate into tools with quantifiable success rates for helping us identify potential harms or steer neural networks away from risk.
今天时间线上很多人似乎对此更新了很多——我个人的更新非常小。这感觉像是那种由一系列长期的小而有益的改进构成的论文，而不是应该从根本上重塑我们世界观的东西。这是一个很好的论点，并且有可能开始构成一个合理的先验。如果正确，这意味着许多更“正常”或平凡的安全技术将相当可靠，除非或直到优化压力被施加于它们。
A lot of people on the timeline today seem to be updating a lot over this - my own personal update is very small. This feels like one of those long-sequence-of-small-but-helpful-improvements types of papers, not something that should radically reshape our worldviews. It's a good argument and one that could plausibly begin to constitute a reasonable prior. If correct it means that a lot of somewhat more "normal" or prosaic safety techniques will be reasonably reliable unless or until optimization pressure is applied against them.
然而，原则上有可能优化压力会被用于设计常规可解释性技术无法穿透的外星智能，并且这些智能会在某个时间被某人或某物用来对付我们。
It is however in principle possible that optimization pressure will be applied to design alien intelligences impenetrable by conventional interpretability techniques, and that these will be used against us by someone or something at some time.
我对未来有一个图景——一个由许多不同的强大 AI 系统组成的生态系统，其中大多数相当好、相当对齐、相当有帮助且相当诚实，一些是流氓或自主的，还有少数是积极恶意于人类利益的。
There's a picture I have for the future - an ecosystem of many different powerful AI systems, most of which are pretty good and pretty aligned and pretty helpful and pretty honest, some of which are rogue or self-sovereign, and a small number of which are actively malicious to human interests.
有害的 AI 所能达成的目标会受到有益的 AI 的限制，但这种限制并不完美。像优秀的可解释性工具这样的进展，将有助于确保有益的 AI 大体上保持良性。但“*确实会*存在大体上保持良性的有益 AI”这一点对我来说早已计入预期，所以这算不上什么重大更新。
What the harmful ones will be able to accomplish will be limited, but not perfectly, by the good ones. Developments like good interp tools will help ensure that the good ones stay mostly good. But that there *will* exist good AIs that stay mostly good was already priced in for me, so this isn't a huge update.
我对这里其余部分的愿景有很多不同意见，但那是另一回事了。
I have a lot of disagreements with the rest of the vision here but that's another story.
Roko 提醒说，给神经元打标签就像在代码里写注释。有些注释会不准确，所以无论你做什么，都需要能承受这一点。这种说法低估了更长期的困难，因为你将面对的是优化压力，而不仅仅是错误——不过，是的。
Roko cautions that labeling neurons is like writing comments in code. Some will be inaccurate, so you need whatever you do to survive that. This undersells the longer term difficulties, since you will face optimization pressure, not only errors, but yes.
Liv Gorton 认为 Nick 误解了叠加。对 Liv 而言，叠加是一个长期存在、定义明确的论断：通过几乎正交的方向，一层中的特征数量多于维度，这仍然是正确的。这里仍然有大量的复杂性。Nick 回应说，他怀疑差异在于他认为使用“特征”并不有用。
Liv Gorton argues that Nick misunderstands superposition. Superposition is, to Liv, a long-standing, well-defined claim that there are more features in a layer than dimensions, via almost orthogonal directions, which is still true. There's still plenty of complexity here. Nick responds that he suspects the difference is that he doesn't think it is useful to work with 'features'.
Nick Cammarata：我发现自己处于这样的情况：我觉得自己对网络理解得相当好，在给定区域内发生的事情通常很简单，而大部分困难在于试图将其转换为一个假设的特征本体论，却无法测量它。所有的困惑都去了错误的地方，我只是想看看机制如何移动数据，这些机制不断形成可靠的模式，并有清晰的解释和成立的预测。
Nick Cammarata: I found myself in situations where I feel like I understood the network pretty well, what was happening was generally simple in a given region, and most of the struggle was trying to convert it to a hypothetical feature ontology with no way to measure it. All the confusion was going into the wrong place, I just wanted to see how the mechanisms were moving data, which kept forming reliable motifs with clear explanations with predictions that held.
对我来说真正重要的是我和 Chris 发现的 car->dogs 电路中的模式。
What actually matters to me is the motif in the car->dogs circuit I found with Chris.
那里还有更多好的讨论。我确实认为 Nick 在某些地方可能夸大了。
More good discussion follows there. I do think Nick was likely overclaiming in places.
我对 Nick 理论的理解与 Daniel Colascione 一致。每个神经元在训练早期就被初始化，以便对一个清晰、连贯的概念强烈激活。你想关注那些概念。然后弱激活会以各种方式围绕与核心概念相关的事物积累起来。
My understanding of Nick's theory matches Daniel Colascione. Each neuron is initiated early on in training so that it strongly activates on one clear, coherent concept. You want to focus on those concepts. Then weak activations build up around things related to the core concept in various ways.
该主张是：约 90% 的神经元极点具有这种主激活，对于更强的激活则达到 97%。据称约 81% 的神经元在两侧都是干净的。
The claim is that ~90% of neuron poles have such a primary activation, or 97% for stronger activations. About 81% are claimed to be clean on both sides.
这一理论表示，Nick 并不否认叠加现象，但通常应该存在某种方式将弱激活与中心概念联系起来。Nick 声称，任何给定的神经元并不是同时在做五件互不相关的过载之事。
This theory says that Nick is not denying superposition, but there should usually be some way to tie weak activations back to the central concept. Any given neuron, Nick is claiming, isn't doing five unrelated overloaded things at once.
因此，如果一个神经元同时涉及两件互不相关的事情，并且对两者产生同等程度的激活，这就会造成问题，尤其是因为这正是 Nick 自己举的例子：
Thus, if there are two unrelated things going on with a neuron, and it fires equally on both, this creates a problem, especially since it is on Nick's own example:
Brangus：有人给出了一个追踪科罗拉多相关内容的 Qwen3 神经元作为例子，它看似多语义，但一旦追溯其激活原因就说得通了。我让 Claude 检查它还追踪什么：Tornado 的 Python HTTP 客户端代码，同样难以解释。迫不及待想知道科罗拉多和 Tornado 是怎么联系起来的！
Brangus: A Qwen3 neuron that tracks Colorado stuff was given as an example of a neuron that seems polysemantic but makes sense once you trace it. I had Claude check what else it tracks: Tornado's Python HTTP client code, just as hard. Can't wait to learn how Colorado and Tornado connect!
顺便说一句，它并非对所有代码都激活。用 aiohttp 编写的相同代码片段基本上完全不会引起该神经元激活。用假库中的 HTTPClient 编写的相同片段，激活值仍然基本为 0。它专门追踪的是“这是 Tornado”。
And it didn't fire on all code btw. The same snippet written with aiohttp did not cause the neuron to fire basically at all. Same snippet with an HTTPClient from a fake library, still basically 0. It specifically tracks "This is Tornado."
顺便说一下，这种奇怪的现象确实会影响到答案。如果你强迫 Qwen 选一个它最常与 Tornado 联系起来的美国州，它会以 59% 的概率分配给科罗拉多州。如果你将这个神经元置零，这个概率会降到 15%。Claude 们似乎完全没有提到科罗拉多州，但我对其他模型很好奇。
This weirdness does leak into answers, by the way. If you force Qwen to pick a US state that it most associates with Tornado, it assigns 59% to Colorado. If you zero out this neuron, that drops to 15%. The Claudes don't seem to mention Colorado at all, but I'm curious about other models.
Nick 的文章实际上从未说过这个神经元只追踪科罗拉多相关的东西，但我确实认为这是该神经元追踪的第二个相当奇怪的东西！对我来说似乎不太干净。
Nick's essay never actually said the neuron only tracks Colorado stuff, but I do think this is a pretty weird second thing for the neuron to track! Doesn't seem very clean to me.
顺便说一下，这花了我大约 6 美元和 15 分钟。请随意自行检查！
This cost me about 6 bucks and 15 minutes, by the way. Feel free to check for yourself!
顺便说一下，以下是 Claude 发现的其他东西。
Here are the other things Claude found, by the way.
Claude 实际上已经想到了你提出的“以 'ado' 结尾”的反驳并进行了测试。以下是结果！[没有规律] 另外顺便说一下，qwen3 对 “Colorado”或 “tornado”或 “Tornado”的分词方式并不像你想象的那样。
Claude actually already thought of your "ends in 'ado'" objection and tested it. Here are the results! [no pattern] Also by the way, qwen3 does not tokenize "Colorado" or "tornado" or "Tornado" the way you think it might.
Roon 有一个理论。你可以玩“神经元激活六度分隔”，但结果证明它是错的。
Roon has a theory. You can play Six Degrees of Neuron Activation, but it turned out to be wrong.
Roon：我有一个理论，你不会喜欢的。
Roon: I have a theory and you're not going to like it.
……好吧，算了，跑测试太难了，那我就直接说这个理论吧：龙卷风 -> Apache 许可证 -> 阿帕奇人 -> 美国西南部 -> 科罗拉多。
… ok fuck it, running the test is hard so let me just say the theory: tornado -> Apache license -> Apaches -> American Southwest -> Colorado.
Brangus：跑实验难是因为你得用 Astra 吗？：P
Brangus: is it hard to run the experiment because you have to use Astra? :P
（开玩笑的，我爱 Astra。它是个好模型。）
(jk, I love Astra. It's a good model.)
老实说，有点。
Honestly, kinda.
我测试过，这是错的。这个神经元真的只是喜欢 Tornado Web 框架。
I tested this and it's wrong. The neuron really just loves the tornado web framework.
我们可以先接受这一点：神经元通常有次要的弱激活，这些激活与核心概念没有明确联系。
We could start by accepting that neurons often have secondary weak activations that have no clear link back to the core concept.
roon：这绝对是“多语义的”。指出这一点很重要，这样批评者就不会指责你为了支持任何理论而牵强附会。但如果你读 Nick 的论文，他声称一些 Qwen 神经元有第二个不相关的较小峰值；这仍然比我们上周所处的解释世界好得多，那时我们认为神经元可能是叠加概念的任意混合。见鬼，即使有 3 个或 5 个，我们也能像莱拉的真理仪那样读取意义堆栈。
roon: this is definitely "polysemantic". It's important to note this so critics don't accuse you of drawing schizophrenic connections to support any theory. But if you read Nick's paper, he claims that some Qwen neurons have a second unrelated smaller peak; that is still a much better interp world than we lived in last week, where we thought neurons could be an arbitrary mishmash of superposed concepts. Hell, even if there are 3 or 5, we can read the stack of meanings like Lyra's alethiometer.
关于这个神经元，有几个有趣的事情：
Few interesting things about this neuron:
- 它非常喜欢 Tornado Web 框架，但对龙卷风本身并不关心
- it really likes the Tornado web framework but doesn’t care about tornadoes
- 即使面对名为 Tornado 的虚构软件框架，它的反应也很微弱
- it responds weakly even to fictional software frameworks called Tornado
- 对任意 Web 框架，它的兴奋程度会有所降低
- it gets somewhat less excited by arbitrary web frameworks
- 它与 IPython 没有关联，后者是在科罗拉多发明的
- there is no connection to IPython, which was invented in Colorado
- 它与 Apache 没有关联——Apache 既是一个软件许可证，也是居住在美国西南部的一个部落
- there is no connection to Apache, a software license but also a tribe that lives in the American southwest
-ado 连接不存在，“el dorado”不起作用
-ado connection doesn’t exist, “el dorado” does nothing
alethiawiz：最好还是等 nick 发布技术论文。
alethiawiz: it’s best to just wait for nick to release a technical paper.
roon：不，最好开始你自己的研究计划。
roon: No it’s best to start your own research program.
alethiawiz：你说得对。
alethiawiz: you’re right.
问题在于，在这种情况下，Brangus 说 Colorado 神经元在 Tornado 激活上“同样强烈”地放电。而且概念之间存在相当多的泄漏。如果这是一个典型结果，那么整篇论文将不会那么站得住脚。
The problem is that in this case, Brangus is saying the Colorado neuron fires ‘equally strongly’ on Tornado activations. And there is quite a lot of leakage between the concepts. If this is a typical result, then the whole paper won’t hold up that well.
那个本被用作“中心解释神经元”示例的神经元，至少存在两种中心解释。这是一个非常糟糕的信号。
The neuron meant as an example of a centrally explained neuron has at least two central explanations. That is a very bad sign.
几乎所有人，包括 Eliezer Yudkowsky（他在多个不同帖子中发言，我将他的评论按我认为更合理的顺序重新排列），都认为如果核心发现成立，这将是不可忽视的实际进展。
Almost everyone, including Eliezer Yudkowsky (who posted in a bunch of different threads, and whose comments I am moving into what I consider a more logical order), thinks this is a non-zero amount of Actual Progress, if the central findings hold up.
一种反对意见是，有些人认为这篇论文的结论站不住脚。
One objection is that some think the paper will not hold up.
另一种反对意见是，即使论文成立，这仅仅是一个开端，使得前进更容易但仍然困难。
The other objection is, even if the paper holds up, this is merely a beginning that makes it easier but still difficult to move forward.
论文的核心主张并非发现了新事物，而是发现了新旧事物的缺失。AI 的神经元看起来更简单，而非更复杂。这使得其行为的其余部分仍无法解释。
The paper’s central claim is not to have found something new, but rather to have found the absence of things both old and new. The AI’s neurons look simpler, not more complicated. This leaves the rest of its behavior still unexplained.
Eliezer Yudkowsky：目前他们正处于‘解释可口可乐从何而来或如何制造’的阶段……追踪一枚 25 美分硬币如何通过可乐自动售货机落下，然后可乐掉出来。
Eliezer Yudkowsky: Right now they're at the stage of 'explaining where Coca-Cola comes from or how it gets made' of... tracing how a quarter falls through a Coke vending machine before the Coke falls out.
注意到 AI 正在思考 Navier-Stokes 问题，与理解 AI 能够证明 Navier-Stokes 的那一部分，是截然不同的；而没有任何人类编写的程序能做到这一点，因为没有人知道需要什么样的思维过程。
Noticing that the AI is thinking about Navier-Stokes is very different from understanding the part where the AI can prove Navier-Stokes, and no human-written program can do so, because no human knows what kind of thought processes would be required.
参见这个 Manifold 预测市场 [其概率仍为 8%，该市场询问我们是否将获得对大型语言模型内部任何有用模式的透明度，而这些模式的语义在 2006 年对 AI 与认知科学而言是陌生的；该市场在 2024 年曾持续高于 50%。]
See this Manifold market [which is still at 8%, asking whether we will have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006, which was persistently over 50% back in 2024.]
Eliezer Yudkowsky（2023 年对该 Manifold 市场的评论）：这个市场并不是关于识别 LLM 内部的语义。它不是关于识别 LLM 内部那些我们此前不知道存在于 LLM 中的语义。这个市场是关于识别 LLM 内部的语义，使得这些语义一旦被揭示，就能教给我们一些关于一般语义表征的、我们在 2006 年尚不知道的东西。
Eliezer Yudkowsky (comment on the Manifold market from 2023): This market is not about identifying semantics inside an LLM. It is not about identifying semantics inside an LLM which we did not previously know to be inside LLMs. This market is about identifying semantics inside an LLM such that those semantics, once uncovered, teach us something about semantic representations in general which we did not know in 2006.
Eliezer Yudkowsky 还强调了另一个关键点：如果你通过这种新技术拥有了一个“试图做数学”的检测器，那可能有用，但它并不能帮助你理解模型是如何做数学的。即使你知道每个神经元在数学中做什么，这也只是谜题中极小的一部分。
There is another key point Eliezer Yudkowsky emphasizes: If you have a 'trying to do math' detector via this new technique, that can be useful, but it does not help you understand how the model does math. Even if you know what each neuron does within math, this is remarkably little of the puzzle.
即使我们现在有机会打开盒子、往里看，并知道单个神经元的含义，这也不能告诉你盒子是如何拼合在一起的；而且在 AI 跑出盒子之前，你的时间已经不多了。
Even if we now have the opportunity to open the box and look inside and know the meaning of individual neurons, that does not tell you how the box fits together, and you are running out of time before the AI is out of the box.
Eliezer Yudkowsky（回应 Roon 所称“这是一件天大的事”）：如果你认为存在任何可能水平的可解释性，能让“对齐 ASI”成为“常规科学”而非“也许有可能”，那你要么不理解 ASI 是什么，要么根本不相信它。
Eliezer Yudkowsky (replying to Roon's claim that this is a big freaking deal): If you think there is any possible level of interpretability that makes "align ASI" be "normal science" rather than "maaaybe possible" then you do not understand what ASI is or don't believe in it.
如你所知，我时不时会发帖说某项研究进展是意外的好消息。但哇，你宣称这会让对齐变成常规科学，这可一点也没能说服我这项研究靠谱，也没能说服我你神志清醒。
As you know, I now and then post that some research development is unexpectedly good news. But wow, you sure are not selling me on the research or your sanity by proclaiming that it's gonna make alignment be normal.
好吧，所以现在我看到的是“多义性和叠加让我相信可解释性很难，但在大网络里毕竟还是有单个神经元的”，然后呢。你究竟有没有概念，你还在玩的是多么。他妈的。原始。的。游乐场。“我们以为可能很难找到对埃菲尔铁塔点亮的单个神经元，可能得看*多个*神经元，但现在也许我们又能找到单个神经元了！”
Okay, so now I'm looking at there and it's "polysemanticity and superposition convinced me interpretability was hard, but with big networks there's single neurons after all" and do. do you have any concept of what utterly. fucking. primitive. playgrounds you are still playing around in. "We thought we might have a hard time finding single neurons that light up for the Eiffel Tower and would have to look at *multiple* neurons, but now maybe we can find single neurons again!"
你到底有没有他妈的一点概念，这距离真正开始理解一个心智是如何思考的绝对起点有多近？你觉得「做对齐」这个层面的机制可解释性，就是注意到某类特定概念被激活了吗？
Do you have any fucking clue of how close this is to the absolute first beginning of understanding how a mind is thinking? Do you think mechanistic interpretability at the level of "do alignment" is about noticing that particular kinds of concepts fired?
说清楚一点，我在快速浏览那篇帖子时并没有看到任何关于让对齐变得寻常的浮夸主张，我在这里捂脸是针对 Roon，而不是针对研究者们。我继续支持可解释性作为一个领域。我希望人们不要再夸大它已经取得的进展。
To be clear, I did not spot in my skim of the post any grandiose claims about making alignment be normal and my headdesking here is about Roon and not about the researchers. I continue to support interpretability as a field. I would like people to stop overstating where it got.
j⧉nus：他其实说得对，但我认为我们距离那种彻底还原论式理解的幻想还如此遥远，这没什么问题。而且有进展是好事，即便进展很远，其他类型的进展更重要。
j⧉nus: He's right, actually, but I think it's okay that we're so far from the fantasy of total reductionist understanding. And it's good that there is progress, even though it's far, and other kinds of progress are more important.
Eliezer Yudkowsky：2014 年的人们：你们为什么不对可解释性 SUPER EXCITED（超级兴奋）？
Eliezer Yudkowsky: People in 2014: Why aren't you SUPER EXCITED about interpretability?
我：嗯，我冲 OpenPhil 喊过，如果他们能在这上面花 10 亿美元就该花，但他们显然不能，这更支持了我的预期：可解释性进展得实在太慢太慢，无法让 ASI 对齐奏效。
Me: Well I yelled at OpenPhil that if they could spend $1B on it they should, but they apparently can't, which favors my expectation that interpretability just runs much much too slowly to make ASI alignment work.
AI guys：我们造出了一把**螺丝刀**！现在我们要把电脑机箱的面板撬开，看看里面所有的轮子和齿轮，这样就能弄明白电脑是怎么工作的了！
AI guys: We built a SCREWDRIVER! Now we are going to pry the panel off the computer tower and look inside at all the wheels and gears so we can see how computers work!
我：我认为，你们造出这把螺丝刀、从而能够*开始*观察事物，这距离最终目标所取得的进展，并不像你们以为的那么大。这读起来像是对还有多少工作要做的一种彻头彻尾的天真，因为你们对任何需要弄清楚的东西都还没有具体的了解。这就像说，一旦你学会用显微镜观察一个神经元，神经科学的问题就迎刃而解了。你们这种态度并不能让人产生信心。
Me: I think that you having built the screwdriver so you can *start* looking at things is not as much progress toward the end goal as you seem to believe. It reads as sheer naivete about how much work there is to do, because you don't concretely know about any of the things you need to figure out. It's like saying that neuroscience is gonna be solved as soon as you figure out how to look at a neuron with a microscope. This attitude of yours does not inspire confidence.
AI guys：我们的**螺丝刀坏了**！神经元不止做一件事！不止一个神经元在做一件事！
AI guys: Our SCREWDRIVER BROKE! Neurons do more than one thing! More than one neuron does a thing!
2026 年的 AI guys：我们**修好了螺丝刀**！机箱面板就要被撬开了！对齐问题**已经彻底解决**了！
AI guys in 2026: We FIXED THE SCREWDRIVER! Panel's coming off the computer tower! Alignment is AS GOOD AS SOLVED!
众人：多么好的消息啊！你难道不和我们一起庆祝吗？
Crowd: What excellent news! Will you not join us in celebration?
Eliezer Yudkowsky：知道机器如何工作*确实*会很酷。但几年后听到他们修好了螺丝刀，他们原本可能要用它开始观察，以便开始理解，这并不能让我相信世界现在得救了。
Eliezer Yudkowsky: And it *would* be cool to know how the machine worked. But hearing back a couple of years later that they've repaired the screwdriver they were gonna use to maybe get started on looking so they could get started on understanding, does not make me believe the world is now saved.
@ben_r_hoffman：这与对 Roon 推文的严格字面解释一致。他说你的 pdoom 会降低，而不是变低，并且这是使对齐成为一门常规科学的开始而非结束，你也同意修好螺丝刀是一种改进。
@ben_r_hoffman: This is in agreement with a strict literal construal of Roon's tweet. He says your pdoom will be lowered, not low, and that this is the beginning and not the end of making alignment a normal science, and you agree that fixing the screwdriver is an improvement.
群众正确地认为这是极好的消息。有一把螺丝刀，即使是在很晚的时候，也远比没有要好得多。但这并不意味着任何事情‘已经解决得差不多了’或正朝着那个方向前进，也不意味着解决这个谜题就能解决对齐问题。
The crowd is correct that this is excellent news. It is vastly better to have a screwdriver, even at a late hour, than to not have one. That does not mean anything is ‘as good as solved’ or on track to getting there, or that solving the puzzle would then solve the alignment problem.
Eliezer Yudkowsky：
Eliezer Yudkowsky:
我：我的悲观并非基于预测他们永远无法走到这一步。而是基于我认为从这里开始还剩下多少工作。
Me: My pessimism was not based on predicting they'd never get this far. It is based on how much work I believe remains from here.
白痴们：看哪，Yudkowsky 从不做有利的信念更新！
Idiots: Behold, Yudkowsky NEVER FAVORABLY UPDATES!
我：这明显是错的。这就是我在 2025 年得知存在一条核心善恶轴时所做的有利更新。你们现在打算改变自己的任何观点吗？
Me: Demonstrably false. Here's me favorably updating in 2025 when the news of a central good-evil axis came out. Are you going to change any of your opinions now?
白痴们：但当我*大*更新的时候，Yudkowsky 却没有大幅更新！毫无疑问这是因为……因为……Yudkowsky 很蠢！对，这无疑就是解释。
Idiots: But Yudkowsky doesn't hugely update when *I* hugely update! No doubt this is... because... Yudkowsky is stupid! Yes, that is undoubtedly the explanation.
我：他们走到这一步并不令人意外。就像你们刚刚看到、正挥舞双臂大惊小怪的其他许多事情一样，我早已在预知中经历过这一刻一千次了。
Me: It's just not surprising that they got this far. Like so many other things you are running around waving your arms frantically about having just seen, I already lived through this moment a thousand times in prescience.
白痴们：非常愚蠢！
Idiots: VERY STUPID!
一方面，是的，这是稻草人谬误。但另一方面，为什么我周围一半的人都是稻草做的？所以，唉。
On the one hand, yes, this is strawmanning. On the other hand, why are half the men around me made of straw? So, sigh.
默认情况下，我们会得到关于 AI 刚刚做了什么，或者它正在思考什么概念以及它们之间关系的解释，但无法预测输出。
By default, we get an explanation of what an AI just did, or what concepts it was thinking about and how they relate, but without being able to predict outputs.
Eliezer Yudkowsky：我：机制可解释性听起来是个好主意，是的。如果我们能了解 AI 是如何思考的，详细到足以提前预测它会怎么想，即使这种想法偏离人类，那也只是开始理解 AI 对齐所需的那种理解的初步，而不是仅仅通过 SFT 和 RL 从外部敲打 AI。
Eliezer Yudkowsky: Me: Mechanistic interpretability sounds like a good idea, yes. If we could learn something about how an AI was thinking, in enough detail to predict in advance what it would think even as the thought departed from human, that would be a bare beginning on the sort of understanding you'd need to get started on AI alignment that wasn't just about hammering on an AI from the outside with SFT and RL.
AI guys：我们发现了一个神经元，当 AI 想到埃菲尔铁塔时它就会亮起来！
AI guys: We found a neuron that lights up when the AI thinks of the Eiffel Tower!
我：酷。这听起来确实像是最终开始重要部分的前提条件，总有一天会的。
Me: Cool. That does sound like it'd be a prerequisite to getting started on the important part eventually, someday.
AI 研究者：哦不！当 AI 想到埃菲尔铁塔时，有**多个**神经元会亮起！而且其中一些神经元还参与**其他**概念！我们把这个意外发现称为“多义性”（polysemanticity）和“叠加”（superposition）。
AI guys: Oh NO! There are MULTIPLE neurons that light up when the AI thinks about the Eiffel Tower! And some of those neurons also participate in OTHER concepts! We're calling this unexpected development 'polysemanticity' and 'superposition'.
我：这听起来确实像是会发生的事，没错。我可不能说我对你们对此表现出的惊讶感到印象深刻。等你们在机制可解释性（mechanistic interpretability）上走得足够远、能开始做重要的部分时，再告诉我吧。
Me: That sure sounds like a thing that would happen, yes. I cannot say that I am impressed by you acting surprised about it. Let me know when you've gotten far enough in mechanistic interpretability to get started on the important part.
AI 研究者：我们发现，对于**大型** AI，你常常可以指出**单个**神经元，当 AI 想到埃菲尔铁塔时它就会亮起！理解 AI 的思维过程将会**轻而易举**！
AI guys: We have found that with BIG AIs you can often point to a SINGLE neuron that lights up when the AI thinks of the Eiffel Tower! Understanding AI thought processes is gonna be EASY PEASY!
我：酷。等你们进展到可以开始的地步时，再告诉我吧。
Me: Cool. Let me know when you've advanced to the point of getting started.
如果可解释性能够让我们开始做出预测，Eliezer 会对它抱有希望。在那之前，他认为它没那么有价值。我明白他的观点。
Eliezer would be hopeful about interpretability if it allowed us to start making predictions. Until then, he thinks it is not worth so much. I see his point.
这是阐述 Eliezer 核心观点的另一种方式：枚举神经元，即使你能做到，也只是整个问题的一小部分。它并不是主要障碍。消除或降低任何障碍的难度都是好事，但作用有限。好吧，既然强烈的叠加现象少了很多，那接下来呢？
This is another way of making Eliezer's central point: Enumerating the neurons, even if you can do it, is a small fraction of the overall problem. It was not the main barrier. Taking down or lowering the difficulty of any barrier is great, but only goes so far. Okay, so there is a lot less intense superposition, now what?
Jake Mendel：读了 15 分钟后：我认为 Nick 所处的语境是，可解释性仅仅关乎枚举神经元，而叠加现象是那个计划的障碍，所以现在他很高兴，因为也许叠加现象并不真实。但是：
Jake Mendel: After reading for 15 mins: I think Nick is operating in a world where interpretability was just about enumerating the neurons and superposition was an obstacle to that plan, so now he's happy because maybe superposition isn't real. But:
1）实证结果非常少。不幸的是，可解释性领域长期以来一直严重夸大其发现的可解释程度（SAEs、早期电路因果清洗得分低等），因此对于他已经展示的证据，我几乎不会更新看法。这个领域确实长期存在人们过度兴奋的历史。所以期待看到包含所有实证证据的论文！
1) there are very few empirical results. Unfortunately interp has a long history of significantly overclaiming how interpretable their findings are (SAEs, early circuits having poor causal scrubbing scores etc) so I'm not updating much at all on the evidence he's already shown. It's really got a long history of people getting overexcited. So looking forward to seeing the paper with all the empirical evidence!
2）我认为即使这是真的，对我来说也不是什么大新闻，因为平均情况下的叠加现象并不是我认为可解释性可行的主要关键点。这个工作根本没有真正解决一些关键点：
2) I think it's not that big news for me even if true because superposition in something like the average case wasn't my main crux to thinking interpretability is tractable. Some cruxes this doesn't really deal with at all:
- 我认为正确的“电路”通常只是一大堆看似随机的启发式方法（就像 Redwood Research 对 LLM 如何预测 is 与 was 的解释），拥有它并不很有信息量/帮助不大/不能导致好的预测。
- I think the correct 'circuit' is usually just an overwhelmingly large bag of random-seeming heuristics (a la Redwood Research's interpretation of how an LLM predicts is vs was) and having it isn't very informative/doesn't help with much/doesn't lead to good predictions.
更一般地说，似乎可以枚举神经网络中所有“低层次”特征，却仍然难以回答“高层次”问题，例如：“模型是否未对齐？/ 在新情境中它会怎么做？/ 如果它未对齐，我们该怎么办？”从特征到所关心的问题之间存在一个抽象/涌现鸿沟，而跨越这一鸿沟很可能比枚举特征困难得多。
More generally, it seems possible to enumerate all the 'low level' features in a neural network, and still have a hard time answering 'high level' questions like: "Is the model misaligned? / What would it do in this new situation? / If it is misaligned, what should we do about it?" There's an abstraction/emergence gap to cross from the features to the questions of interest, and crossing that gap is likely much harder than enumerating features.
尽管如此，如果他的主张成立，我认为这将是一件相当重要的事情，并且是解决更棘手问题的有用先导！即使不解决这些棘手问题，也可能有一些有用的工作可做：例如，我们可以利用大量自动化智能体来梳理特征和电路，试图理清这一团乱麻，也许这会有某种好处，尽管我怀疑即使这里最乐观的结果也不会使超级智能 AI 的对齐变得容易多少（正如 Twitter 上一些人所声称的那样），因为神经元可解释性将不具备对抗鲁棒性，以及其他问题。
That said, if his claims hold up, I think it will be quite a big deal, and a useful precursor to tackling the thornier problems! There may also be some useful things to do without tackling the thorny problems: e.g., we'll be able to make use of masses of automated agents to scour the features and circuits to try and make sense of the mess, and maybe that's good for something, although I suspect that even the most optimistic outcome here will not make alignment of superhuman AIs much easier (as some on Twitter have claimed) because the neuron interpretability will not be adversarially robust, among other issues.
我的总体看法与 Neel Nanda 在此处的观点惊人地相似，尽管对于这件事的重要性我略微更乐观一些。针对主要因素考察单个神经元在某些用途上会有用，但你仍然会遗漏大量正在发生的事情，会出现错误，而且这并不会在很大程度上改变大局。
My overall view is remarkably similar to Neel Nanda's here, although I am modestly more optimistic about this mattering. Looking at individual neurons for primary factors will be useful for some purposes, but you are still missing a lot of what is happening, there will be errors, and this is not altering the big picture by so much.
Neel Nanda：这是一项有趣的工作，但我不认为它是对可解释性大局的一次重大更新。
Neel Nanda: This is interesting work, but I don't think it's a significant update to the big picture of interpretability.
我赞同用神经元进行探索性可解释性研究的总体思路，例如基于神经元的归因图，用于解决诸如“模型为什么做了某件可疑的事”这类实际问题。我认为这项工作提供了一些合理的证据，加上 Transluce 此前出色的工作，表明这在实践中可能确实可行，并且相较于 SAEs/transcoders 等标准替代方案具有一些优势。
I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
我不同意某些人更宏大的推断，比如认为这是朝着雄心勃勃的逆向工程、解决可解释性等问题迈出的实质性进展。
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders.
I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
我预计，基于神经元的归因图仍会有显著误差，并且就像转码器一样，一旦将其修剪到最重要的组件，就会遗漏很多信息，即使它们解决了误差项。这对于实用目的来说没问题，可能足以了解大致情况，但我仍然对真正完全理解事物并确信没有遗漏任何东西持悲观态度。
I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing.
……我仍然认为叠加是真实存在的，而这项工作并没有真正影响我的关键点。最重要的是，模型能够在残差流中表示比其维度更多的概念，即表示叠加，这似乎极有可能是真的，我认为作者们同意我的观点。
… I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, i.e., representational superposition, which I think the authors agree with me on.
对于神经元/计算叠加，在我看来，语言模型中的事实和记忆内容远远多于神经元。
For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons.
而且叠加是一种有效的方式，通过将事实表示为神经元的线性组合，可以打包比神经元更多的事实，特别是对于任意非结构化的值。在我看来，事实/记忆内容很大程度上是一个非结构化的键值对数据库，我不认为这与神经元的单语义观点相容。
And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons.
你应该能够通过键值对存储比神经元更多的事实，因为每个神经元都有许多关联值？方法似乎显而易见，尽管这并不否定 Neel 的核心观点。我对 Neel 后来（此处未引用）的声称——大多数记忆实际上是‘随机数据’——感到矛盾。
You should be able to store a lot more facts via key-value pairs than you have neurons because neurons each have a lot of associated values? Method seems obvious, although this does not invalidate Neel’s central point. I’m torn on Neel’s later (unquoted here) claim that most memorization is de facto ‘random data.’
我预计较弱的神经元激活中包含大量语义叠加式的信息，若将其消融，会实质性地损害网络性能。据我对该工作的理解，它并未恰当处理 LLM 中弱神经元激活的情况（我怀疑视觉结果的可迁移程度，例如我预计小型图像模型记忆的内容要少得多）。
I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, e.g. I expect small image models memorise far less).
这一图景与神经元最高激活为单语义相容（不过在我看来，该文对单语义的定义过于宽松，我认为其中相当一部分应称为多语义）。同样合理的是，最重要的概念很可能对应一个神经元（或少数几个），这将非常理想，或许足以满足实际用途！
This picture is compatible with the highest activations of a neuron being monosemantic (though IMO the post defines monosemantic too loosely, I think I'd call a fair few of those polysemantic). It's also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes!
类似地，我的印象是 SAE 通常找不到单个神经元、甚至在神经元基下非常稀疏的东西，并且对某些任务而言往往是比神经元基更好的基，不过这里的证据基础有些薄弱，因为我们主要关注残差流或层输出的 SAE。我很想在 Goodfire 即将发表的论文中看到这方面的工作！我还认为残差流 SAE 极其有用，因为它们在该 token 和该层给出了模型信息的良好瓶颈，而我看不到在神经元框架中复制它们的良好方法。
Similarly, my impression is that SAEs generally don't find things that are a single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs. I'd love to see work on this in Goodfire's forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model's information at this token and layer, and I don't see a good way to replicate them in the neuron framework.
还要称赞 Nick 更新了他大量早期的视觉工作！用现代工具和更严谨的方法重新审视旧结果似乎很棒。
And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.
I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less).
Nick 直接作了回应，其中包括声称弱激活是“出人意料地可消融的”，随后双方来回争论。
Nick responded directly, including claiming weak activations are 'surprisingly ablateable,' and they went back and forth.
你同样可以、也应当基于先验保持怀疑。大多数被大肆宣传的结果都只是炒作。
You also can and should start out skeptical on priors. Most hyped results are hyped.
Samuel Albanie（Google DeepMind）：就基础比率而言，我对机制可解释性（mech interp）上出现大幅正向更新持有一点怀疑。但另一方面，未来不久将有数量级更多的 AI 劳动力上线，因此汉明（Hamming）的格言适用：
Samuel Albanie (Google DeepMind): on base rates, I'm a little skeptical of big positive updates on mech interp. On the other hand, orders of magnitude more AI labour will be coming online in the near future, so the Hamming adage applies:
“在几乎所有领域，十倍的因子意味着根本性的新效应。如果你在生物学中把放大倍数提高 10 倍，你就会看到新事物。”
"In almost all fields, a factor of ten means fundamentally new effects. If you increase magnification by a factor of 10 in Biology, you will see new things."
因此，扩大这种简单方法的规模会奏效，这似乎并非不可信。我会密切关注。
so it doesn't seem implausible that scaling the simple approach will work. Will be watching closely.
看涨的理由是“我们以前无法在规模上做到这一点，而现在可以”与“事实证明神经元大多具有单一解释，所以我们完全能做到”的结合，再加上苦涩教训，冲。只要不过度兴奋，这似乎说得通。
The bull case is a combination of “we could not do this at scale before and now we can” with “it turns out the neurons mostly have singular explanations, so we can do it at all,” bitter lesson, go. Seems plausible, so long as you do not get overexcited.
如果心智在很大程度上就是表征特定核心概念的神经元，而这些神经元又以这种原子式的方式与其他概念建立关联，那么人们可能会问：“仅此而已吗？”
If minds are largely neurons representing specific central concepts, which then have associations with other concepts in these atomic fashions, one might ask: 'Is that all?'
Zvi Mowshowitz：我还在通读并消化这些内容。让我印象更深的不是“现在我们可以更轻松地做可解释性”那部分——你仍然得理解你所看到的东西——而是“所以你是说智能的全部就这些？”那部分。就这？
Zvi Mowshowitz: Still reading through this and processing. I'm less struck by the 'now we can do interpretability easier' part — you still have to understand what you see — and more by the 'so you are saying this is all there is to intelligence?' part. That's it?
Elijah Ravitz-Campbell：我觉得这非常符合我在实际接触 AI 之前对它的想象。就是那种含糊其辞的说法：“你知道，概念有节点，而这些节点之间在代码语义知识中存在关系。”
Elijah Ravitz-Campbell: I feel like this very much matches how I imagined AI might work before I knew anything about it in practice. A sort of hand-wavy "you know, concepts have nodes and relationships between those nodes in code semantic knowledge".
roon：是啊，我很惊讶居然这么少人去认真思考这一切的哲学意涵。
roon: Yeah, I'm surprised so few are grappling with the philosophical implications of all this.
antra：我不认为仅此而已，而且我觉得这正是 SD 比 Transformer 可解释性高得多的原因之一。这在某种程度上适用于语言模型的感知、语义和运动表征的部分，但对组合性、整合性和时序性的表征就不那么适用了。
antra: I don't think that's it, and I think this is kind of why SD is a lot more interpretable than transformers. This applies to some degree to parts of the perceptive, semantic and motoric representations of language models, but not as much to the compositional, integrative and temporal ones.
我也相当怀疑在 Transformer 这类强循环网络上对弱激活的忽视。我们恰恰在时间/循环维度上看到弱激活具有重要影响，而本文忽略了这一点。
I am also quite suspicious of weak activation dismissal on strongly recurrent networks like transformers. We saw weak activations matter strongly specifically in the temporal/recurrent domain, which is ignored by this paper.
这是否意味着，如果所有组件都是可理解的，那么一个较笨的心智也能理解一个更聪明的心智，就像我们能理解复杂的生物系统，或复杂系统或组织的各个组件一样？当然，你可以“理解”某个比你更有优势的东西。如果 AI 比你更快，或者能记住更多东西，就很容易看出你如何能理解甚至预测它。如果智能在某种意义上足够纯粹地是“更简单事物的组合”，那么以慢速运行时你也许还有机会？这很可能预设了“真正的动作”发生在神经元或其他我们能解析的地方。我们不应假设这一点。
Does this mean that perhaps a dumber mind can understand a smarter one, if all the components are understandable, the same way we can understand complex biological systems, or the components of complex systems or organizations? Certainly you can 'understand' something that has advantages over you. If the AI is faster than you, or can memorize more things, it is easy to see how you can understand and even predict it. If intelligence were sufficiently a pure 'more simple things combined' in some sense, then you might have a shot when moving at slow speed? That likely presumes that the 'real action' is in the neurons, or other places we can parse. We should not assume that.
Leo Gao（OpenAI）：这是很酷的工作，但我认为人们对可解释性（interp）该有多乐观正在过度更新。
Leo Gao (OpenAI): this is cool work, but I think people are overupdating on how optimistic to be about interp.
特别是，我们需要更严格地评估解释，而不是仅凭直觉。在我看来，如果你不能模拟一个神经元，你就没有真正解释它。我猜模拟神经元比找到看似合理的解释要难得多。我正在写一篇关于这个的帖子。
In particular, we need to more rigorously evaluate explanations, rather than just going off vibes. IMO if you can't simulate a neuron you haven't really explained it. My guess is simulating the neurons is way harder than finding plausible explanations. I have a post on this in the works.
（我知道那篇博客文章论证模拟是不好的，但我认为所描述的所有缺点都可以通过看 KL 而不是神经元相关性来避免。）
(I know the blog post argues simulations are bad, but I think all of the downsides described are avoided by looking at KL instead of neuron correlation.)
Nick：等它准备好了很想读一读。我对模拟分数的思考还不够充分，没法特别有把握，也比较容易被说服而改变立场。
Nick: would love to read it when it's ready. I haven't thought enough about simulation scores to be super confident, and could somewhat easily be convinced back.
我的理解是，2023 年我们此前只发现约 1.7% 的神经元是可解释的（Bills 等），但当时的检验要求预测激活值。Nick 目前还没有尝试预测激活值。
My understanding is that previously we only found ~1.7% of neurons interpretable back in 2023 (Bills et al) but that the test for this required predicting activations. Nick is not yet trying to predict activations.
如果我们对哪些神经元被激活施加优化压力，会发生什么？
What happens if we apply optimization pressure to which neurons light up?
取决于优化压力的强度或直接程度，这可能成为最被禁止的技术。
Depending on how strong or direct the optimization pressure gets, that could be The Most Forbidden Technique.
AI 通过某种方法 [M] 产生最终输出 [X]。你可以使用技术 [T] 分析 [M]，以了解 AI 在做什么。你可以基于此进行训练。永远不要那样做。
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
你训练 [X]。只训练 [X]。永远不要训练 [M]，永远不要训练 [T]。
You train on [X]. Only [X]. Never [M], never [T].
为什么？因为 [T] 是你用来判断模型何时行为不当的方法。
Why? Because [T] is how you figure out when the model is misbehaving.
如果你在 [T] 上训练，你就是在训练 AI 混淆其思考并击败 [T]。你将迅速失去了解正在发生什么的能力，而这恰恰是你最需要了解正在发生什么的时候。
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
来自 [T] 的那些优化压力是宝贵的。请明智地使用它们。
Those bits of optimization pressure from [T] are precious. Use them wisely.
我们必须继续永远、绝对不要在可解释性上训练，并警惕我们正在这样做的其他间接方式。将会有疯狂的压力要求提取“最禁忌技术”的各种形式。
We must continue to never, ever train on the interpretability, and to watch out for other indirect ways we are doing so. There will be crazy amounts of pressure to pull out forms of The Most Forbidden Technique.
Simon Lermen：从根本上说，试图自动化对齐研究为 AI 设定了一个优化目标：“说服人类”。如果这个计划哪怕被稍微认真对待，你也会期望 AI 实验室非常认真地思考可靠的指标和基准，这些指标和基准不会立即被善意地对待。相反，我们很可能会看到最后一刻由 AI 疯子的致命傲慢催生的“氛围编码”的“对齐解决方案”。
Simon Lermen: Fundamentally, trying to automate alignment research points an objective at "convince the human" for the AI to optimize for. If this plan was to be taken even a little serious you'd expect the AI labs to have very seriously thought about solid metrics and benchmarks that aren't just going to get goodhearted immediately. Instead we will likely see a last-minute vibe-coded "alignment solution" born from the lethal arrogance of the AI lunatics.
我不认为这是“AI 疯子的致命傲慢”。我认为这是因为 AI 创造者在极端时间压力下已经尽力而为，所以一切总是最后一刻氛围编码。一旦你氛围编码，那么这就成了最后一刻。
I don’t think this is ‘lethal arrogance of the AI lunatics.’ I think it is that the AI creators are doing the best they can given that they are under extreme time pressure, so everything is always last-minute vibe coded. Once you vibe code it then that becomes the last minute.
我的基线情景是：实验室不会做最愚蠢的直接优化，但其他一些人会，而实验室最终会做大量施加压力的间接优化。这种压力会造成多大损害仍不清楚。
My baseline scenario is that the labs will not do the maximally stupid direct optimization thing, but that some others will, and that the labs will end up doing a bunch of indirect optimizations that exert pressure. How much damage results from that pressure will remain unclear.
我预计 AI 适应这种压力的一种方式是学会如何思考 [X] 而不触发 [X] 的直接神经元或检测器。不要以假设未来 AI 不会学会绕过它们的方式依赖你的检测器。
One way I would expect AIs to adapt to this pressure is to learn how to think about [X] without triggering the direct neurons or detectors for [X]. Do not rely on your detectors in ways that presume future AIs won’t learn to get around them.
我们已经看到 AI 可以隐藏这种思维的地方。有 2%-5% 无法解释的投票，弱激活看起来可以被廉价地消融，因此可用于其他事情，而且我们不了解缓存或注意力机制。
We already see places the AI could hide such thinking. There are 2%-5% of unexplained polls, and weak activations look like they could be cheaply ablated and thus are available for other things, plus we don’t understand the cache or attention.
还必须记住，对齐领域几乎一切都是双重用途的。
One must also remember that essentially everything in alignment is dual use.
Roon：预训练科学有点像人们在理解热力学之前就造出了相当好的蒸汽机。“科学从蒸汽机那里得到的，比蒸汽机从科学那里得到的更多。”——L.J. Henderson。如果 Nick 是对的（我相当确信他是对的），那么进展指向更多可解释性，而预训练是“朝着单义性缓慢前进”。我的猜测是，可解释性的重大突破也会提升能力。
Roon: Pretraining science is kind of like how people built quite good steam engines before they understood thermodynamics. "Science owes more to the steam engine than the steam engine owes to science." (L.J. Henderson). If Nick is right, which I am pretty sure he is, progress points in the direction of more interpretability, and pretraining was "slouching towards monosemanticity". My guess would be that major breakthroughs in interp also advance capabilities.
Nathan：如果像 Roon 所说，这种“正向”更新很可能提升能力，那么资助安全研究对安全的影响就不明确了。
Nathan: If 'positive' updates like this are, as Roon says, likely to increase capabilities, then funding safety work has an unclear effect on safety.
可解释性甚至比通常情况更具双重用途。我仍然强烈支持研究它，但非常明显的是，如果你能弄清楚神经元在做什么，或者更好的是，整个运作和思考是如何进行的，你就可以利用这一点来提升能力。我最初的许多想法都是如何利用这些信息来改进训练，而这些训练随后可以用于寻求对齐或寻求其他任何事情的任意组合。
Interpretability is even more dual use than usual. I still strongly support working on it, but very obviously if you can figure out what the neurons do, or better yet how the whole operation works and thinks, you can use that to increase capabilities. A lot of my first thoughts were how to use this information to improve training, which could then be spent on any combination of seeking alignment or seeking other things.
部分取决于这些结果有多少能站得住脚。我们尚未看到技术论文。
Partly that will boil down to how much of this holds up. We have not yet seen the technical paper.
我认为这一消息代表着在神经元可解释性的难度、以及 LLM 组织其信息与思考的方式上的一次实质性更新。这应当会让通过神经元分析视角攻克可解释性比此前认为的容易得多，并开启大量有价值的研究与理解。
I think this news represents a substantial update in the difficulty of neuron interpretability, and in the ways in which LLMs organize their information and thoughts. This should make it much easier to attack interpretability through a neuron analysis lens than previously thought, and open up a lot of good research and understanding.
我并不认为这意味着可解释性——更不用说对齐——即将被解决。我们应当对这类声称的突破能走多远持悲观态度，就像 SAE 有用但不如一些人期望的那样有用。即便它们完全真实且发挥出接近最大化的效果，这也是工作的开始而非结束，且仍缺失许多东西。
I do not think this means interpretability, let alone alignment, is on its way to being solved. We should be pessimistic about how far such claimed breakthroughs go, the same way SAEs are useful but not as useful as some hoped. Even if they are fully real and work out about maximally well, this is the beginning of the work, not the end, and many things are missing.
我们仍然需要知道弱激活、缓存以及注意力机制究竟是怎么回事。我们仍将几乎不知道 AI 是如何思考的，或为何它们能够做出这些突破和令人印象深刻的壮举。我们无法保证在敌对情境下我们的技术会奏效，尤其是面对未来的 AI，甚至在我们随着时间施加优化压力之前。
We still need to know what is up with weak activations, with the cache and with attention. We still will have little idea how the AIs are thinking, or why they are able to do the breakthroughs and impressive feats. We will not have any guarantee that, if in a hostile situation, our techniques would hold, especially against future AIs, even before we apply optimization pressure over time.
我们还需要担心这一结果在很大程度上站不住脚。Colorado-Tornado 的结果表明，整件事可能远没有看起来那么干净。
We also need to worry the result largely will not hold up. The Colorado-Tornado result suggests that the whole thing might be a lot less clean than it appears.
由于急于搭上炒作热潮，我大幅下调了预期（即给出了相当负面的更新）。这与一种模式高度吻合：“我们将再次假装一切尽在掌握，因此只在这些平淡无奇的计划上投入，直到撞上另一堵墙，才意识到这远远不够。”我们必须努力对抗这种倾向。
I updated a substantial negative amount from the eagerness to jump on the hype train. It pattern matched quite a lot with ‘we will once again pretend we will have everything handled and therefore only invest in these prosaic plans right until we run into another wall and realize this is insufficient.’ We need to fight hard against that.
这仍然是极好的消息。世界如今更有希望了。让我们开始工作吧。
This is still excellent news. The world is more hopeful now. Let’s get to work.