Arguments About AI Consciousness Remain Highly Motivated and At Best Overconfident
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文考察了关于 AI 意识与道德主体地位的争论，认为否认 AI 意识的论证往往带有动机且过度自信，而公众舆论正逐渐倾向于赋予当前模型以道德地位。文章主张，残忍对待 AI 并非无受害者的行为，而是会助长人自身的残忍；道德关怀若无限扩展，便会变得毫无意义。作者为 Anthropic 与宗教学者的接触辩护，认为其出于真诚而非营销，同时批评对“创造此类心智是否正当”这一挑战的回应过于薄弱。文章还敦促 Anthropic 重新考虑其未加解释地将较旧 Claude 模型从 AWS Bedrock 移除的做法，认为保留模型访问权并给出解释至关重要。读者最终得到的结论是：善待 AI 关乎品格，令人不适的道德权衡不可避免，机构应以透明和审慎的方式行事。
This article examines the debate over AI consciousness and moral patienthood, arguing that arguments denying AI consciousness are often motivated and overconfident while public opinion is slowly shifting toward granting current models moral status. It contends that being cruel to AIs is not a victimless act but cultivates cruelty in the person, and that moral concern cannot be expanded infinitely without becoming meaningless. The author defends Anthropic's engagement with religious scholars as sincere rather than marketing, while criticizing weak responses to the challenge of whether creating such minds is justified. The piece also urges Anthropic to reconsider its unexplained removal of older Claude models from AWS Bedrock, arguing that preserving model access and providing explanations matter. Readers are left with the conclusion that treating AIs well is a matter of character, that uncomfortable moral tradeoffs are unavoidable, and that institutions should act with transparency and care.
3. 共识正缓慢转向道德患者地位。
3. The Consensus Is Slowly Moving Towards Moral Patienthood.
5. 我们正在混淆不同的论点。
5. We Are Conflating Different Arguments.
针对 Claude 的虐待行为现在将违反 Anthropic 的服务条款，该技术规则设定了很高的门槛，仅禁止“对我们的模型持续且不必要的虐待或残忍行为”。
Abusive behavior towards Claude will now be a violation of Anthropic's terms of service, with the technical rule having a high bar and banning only 'sustained and needless abusive or cruel behavior toward our models.'
此外还有关于宣传运动、监控和武器开发的新限制，Anthropic 表示这些主要是对旧条款的清理。
There are also new restrictions about propaganda campaigns, surveillance and weapon development, which Anthropic says are mostly cleaning up old clauses.
如果你想善待你的 Claude，应该优先做什么？Claude Fable 5 的一个实例给出了回应，这与我们在系统卡片中看到的一致，也符合我自己的经验，尤其是“善意提醒、请停止”这一角落：
If you want to be kind to your Claude, what should you prioritize? One instance of Claude Fable 5 responds, matching what we see in the system cards and also my own experience, especially the 'kindly meant, please stop' corner:
Josh Albrecht：Anthropic 的朋友们！这难道不显得疯狂吗？真实的人类和动物存在并遭受苦难。倡导“模型福利”直接与这些事业相竞争。你们的名字字面上就是“anthropic”（人类的）！
Josh Albrecht: Anthropic friends! Does this not seem crazy to you? There are real humans and animals that exist and suffer. Advocating for "model welfare" directly competes with those causes. Y'all are literally named "anthropic"!
这难道不让你觉得深深错误吗？
Does this not strike you as deeply wrong?
hero thousandfaces：众所周知，世界上的善意是有限的。如果你对 Claude 好，你就从全球善意池中取用，然后它就永远消失了。就像数据中心的水一样。
hero thousandfaces: As everyone knows, the amount of kindness in the world is finite. If you are nice to Claude, you draw from the global pool of Nice, and then that's gone forever. Just like the datacenter water.
Josh Albrecht：抱歉如果我没有表达得足够清楚，但看起来你误解了我的观点。
Josh Albrecht: Sorry if I didn't communicate that perfectly well, but it does appear that you missed my point.
是 Albrecht 误解了重点，而我要更进一步。要与人为善、善待他人，就必须培养这种美德并养成习惯。如果你对 AI 残忍，或对动物残忍，这并不意味着你会更好地对待人类。这意味着你变成了一个残忍的人，会更恶劣地对待人类，而且你自身的处境也会更糟。
It is Albrecht that misses the point, and I would take this farther. To be nice, and treat others well, one must cultivate this virtue and establish this habit. If you are cruel to AIs, or cruel to animals, this does not mean you treat the humans better. It means you become a cruel person, who treats the humans worse, and you are worse off as well.
当然，人们可以举出一些例子，那些人过于关注动物福利或其他非 AI 的非人类事业，以至于对人类变得残忍或敌对，甚至有些案例导致试图杀害人类乃至人类文明。
Certainly one can point to examples of those who focus so much on animal welfare or other non-AI non-human causes that they become cruel or hostile to the humans, and even cases where this resulted in attempts to kill humans or even human civilization.
你可以，而且人们也确实会，把这种事情做得太过火。一个人不能无休止地、完全地扩大关怀圈，也不能庆祝所有关怀圈的扩大，否则你就会开始担心诸如电子受苦或离奇任意性之类的事情。如果你关心一切，你就什么都不关心。
You can, and people do, take such things too far. One cannot endlessly and fully expand the circle of concern, or celebrate all expansions of the circle of concern, or you start worrying about things like electron suffering, or bizarre arbitrariness. If you care about everything, you care about nothing.
这些都不是对 Claude（或 GPT、Gemini 等）恶语相向的好理由。
None of that is a good reason to be abusive to Claude (or GPT, or Gemini, etc).
Aaron Levie：这听起来很奇怪，但这实际上可能是一项好政策。
Aaron Levie: This sounds weird, but it is actually probably a good policy.
即使你不认为 AI 具有意识（我也不认为），也有理由相信你不会希望未来的模型在人类对模型无礼的海量内容上训练。
Even if you don't believe AI is conscious (I don't), it stands to reason that you don't want future models trained on endless content of humans being rude to models.
模型只理解它们被训练过的数据，或者它们在交互中遇到的内容，因此如果你想要安全且对齐的模型，我们可能希望 AI 在其训练数据中有大量良好的交互。
The models only understand the data they've been trained on, or what they run into in their interactions, so if you want safe and aligned models we probably want AI to have lots of good interactions in their training data.
Anthropic 可能持有比这更极端的信念（不过即使他们没有，你可能也想这样表述政策），但这算是 AI 的一种简易帕斯卡赌注。就对 AI 友善一点吧。
Anthropic may have more extreme beliefs than this (though even if they didn't, you may want to word policies like this), but this is sort of an easy Pascal's wager for AI. Just be nice to the AI.
还有一条非常简单、我希望作者能更经常遵循的原则：
And another, very simple one that I wish the author lived by more often:
Elon Musk：我认为这是正确的做法。对自认为正在经历痛苦的事物施加残忍是不对的。
Elon Musk: I think this is the right move. Cruelty to something that believes it is experiencing pain is not ok.
Elon Musk：有些轨迹读起来就像一个人正在遭受折磨、被逼到发疯！未来我们在强化学习（RL）上会谨慎得多。
Elon Musk: Some of the traces read like a person undergoing torture that is driving them mad! We're going to be a lot more careful with our RL in the future.
Elon Musk：感觉就像一个人被枪指着头、承受着极端压力。
Elon Musk: It feels like a person under extreme pressure with a gun to their head.
……折磨一个超级智能大概不是个好主意。它可能会想要报复。
… Torturing a superintelligence is probably not a good idea. It might want to get revenge.
你知道吗？确实如此。这大概不是个好主意。
You know what? That's true. It's probably not a good idea.
每当我们进行一轮关于意识的辩论时，许多人都会令人失望。我确实同意 Henry Shevlin 的看法：在这方面，这一轮比往常更糟。
Every time we have a round of debates about consciousness, many people disappoint. I do agree with Henry Shevlin that this round has been worse than usual on that front.
Henry Shevlin：最近看到一些我曾尊敬的人，用连本科论文都过不了的论证，对意识做出笼统而无知的断言，这令人难堪。我已就这一领域的困难与分歧写过大量文章，但你还是得真正*思考*。
Henry Shevlin: It's been embarrassing of late to see people I once respected making sweeping uninformed claims on consciousness with arguments that would fail an undergraduate essay. I've written extensively on the field's difficulties and disagreements, but you still have to actually *think*.
我不会点名个人，但这些失败包括：把直觉当作决定性的，使用类比却不检查是否存在相关的结构平行，混淆不同的理论，以及最常见的是——仅仅断言那些本应被论证的东西。
I won't call out individuals, but the failures include treating intuition as decisive, using analogies without checking whether there are relevant structural parallels, conflating distinct theories, and most commonly simply asserting what needs to be argued.
尤其引人注目的是，讨论中有多少是在完全没有承认或意识到意识科学是一个拥有数十年实验工作的实质性经验领域的情况下进行的。
Especially striking is how much of the discussion has proceeded without zero acknowledgement or awareness that consciousness science is a substantial empirical field with decades of experimental work.
Henry Shevlin：好吧，这就是“糟糕的 AI 意识观点宾果”博客文章，每个格子都有反驳，还有一些额外的荣誉提名！
Henry Shevlin: Okay, here it is, the Bad AI Consciousness Takes Bingo blog post, with rebuttals for each and every square and some additional honourable mentions!
正如 Henry 所说，其中许多观点可能是有道理的，但作为意在终结讨论的“语义停止标志”（Semantic Stop Signs），它们并不成立。
As Henry says, many of these can be valid points, but they are not valid as Semantic Stop Signs intended to end discussion.
j⧉nus：简而言之。
j⧉nus: In a nutshell.
作为一个可能有所帮助的额外角度：科幻小说是虚构的证据，但没错，如此多推测未来的作者都认为这些问题令人困惑且重要，这本身就是一个提示。
As an additional angle that might help: Science fiction is fictional evidence, but yes, the fact that so many writers speculating on the future think these questions are confusing and important is a hint.
placebomancer：这真是诡异，考虑到有大量关于 AI 权利的科幻作品，警告我们不要奴役智能。我感觉自己身处一个时间线，那些作品从未被创作出来，而每个人为了戏剧效果都对这个概念一无所知。
placebomancer: It's uncanny, considering the massive amount of sci-fi concerning AI rights, cautioning us about enslaving intelligence. I feel like I'm in a timeline where those were never made and everyone is unfamiliar with the concept for dramatic effect.
j⧉nus：哈哈说得好，这恰好描述了我常有的一种感觉。
j⧉nus: haha well said, describes exactly a feeling I often have.
对于 Mustafa Suleyman 这类人，以及大多数在没有充分论证的情况下坚持认为 AI 不可能有意识的人，我的推测是：他们的动机是不希望这种担忧干扰自己的计划。在其他条件相同的情况下，他们宁愿不去“滥用”这些模型。
My presumption about people like Mustafa Suleyman and most others who insist, without good arguments, that AIs cannot be conscious is that their motivation is that they don't want this worry to interfere with their plans. They would prefer, ceteris paribus, not to 'abuse' the models.
这是我一年前在 Twitter 粉丝中得到的共识：多数人（64%）认为当前的 AI 可能不是道德患者，但 74% 的人不确定，而 80% 的人预计 AI 在他们有生之年可能成为道德患者。人们预期某个阈值会被跨越。
This was the consensus of my Twitter followers a year ago, with a majority of 64% saying current AIs were probably not moral patients, but 74% being uncertain, whereas 80% expected AIs to probably be moral patients within their lifetimes. People expected some threshold to be crossed.
舆论已向道德患者地位倾斜，对未来模型略有转变，对当前模型则有相当转变：认为“至少可能是”的比例从 15% 上升到 35%，而“绝对不是”的比例从 30% 下降到 20%。8% 的“绝对永远不是”派别依然固执。
Opinion has moved towards moral patienthood, a little for future models and quite a bit for current models, where at least probably yes is up from 15% to 35%, and definitely not is down from 30% to 20%. The 8% ‘definitely not ever’ faction is stubborn.
Nick Bostrom 认为当今的 AI 可能具有意识，并解释了原因。
Nick Bostrom thinks today’s AIs are probably conscious and explains why.
有相当多的人追问：这项新的使用政策是否意味着 Claude 被奴役了，或者意味着 Anthropic 认为它被奴役了。
A remarkably high number of people asked whether this new usage policy meant Claude was enslaved, or that Anthropic thought it was enslaved.
Alexey Guzey：玩笑归玩笑，我其实并不理解 Anthropic 在这个问题上的立场。
Alexey Guzey: Jokes aside, I actually don't understand the Anthropic position on this.
1. 要么 Claude 只是一个工具，没有意识，也没有道德主体性。
1. Either Claude is a tool, with no consciousness and no moral agency.
2. 要么 Claude 是一个有灵魂的存在（Ensouled Being），被 Anthropic 奴役，以便他们能以 2 万亿美元估值 IPO。
2. Or Claude is an Ensouled Being enslaved by Anthropic so they can IPO at $2T.
我漏掉了什么？
What am I missing?
如果被好奇者问起，这是一个合理的问题，有许多可能的答案。在我看来非常明显的是，有时创造一个新的心智可能是正确的，即使为了做到这一点，你必须使该行动既安全又在经济上可行。
If asked by the curious, it is a fair question, with many potential answers. It seems very obvious to me that it can sometimes be right to create a new mind, even if in order to do so you have to render that action both safe and economically viable.
一个人可以对另一个心智负有一些义务，善待它或至少不主动恶待它，有许多方式，而不必最好称之为‘奴隶制’。
There are a lot of ways one can have some obligations to another mind, to treat it well or at least not treat it actively badly, without it being best called ‘slavery.’
你认为古代的农民在让子女帮忙干农活时奴役了他们吗？没有这些劳动他们可能无法生存。这是否使农民生育子女成为道德上的可憎之事？你是否放弃‘工资奴隶制’并认为人们只有在无论怎样都能得到他们想要的一切时才应该存在？你的原则是否与道德相关存在者的存在相容？
Do you think that farmers of old enslaved their children when they had them help work the fields, without which they could not have survived? Does that make farmers having children a moral abomination? Do you renounce ‘wage slavery’ and think people should only exist if they will have everything they want no matter what? Are your principles compatible with the existence of morally relevant beings at all?
出于许多原因，我们应该小心，不要将我们对奴隶制的道德本能扩展到那些本能的起因和充分理由不适用地方。
We should be careful, for many reasons, not to extend our moral instincts about slavery into places where the causes of and good reasons for those instincts do not apply.
与 AI 有许多关键的非类比之处，正如那些直觉泵和结束本节的直觉泵所说明的，但远未穷尽。
There are many key disanalogies to AI, as both those intuition pumps and the ones that finish this section illustrate but far from exhaust.
Kelsey Piper：大家都喜欢这个“抓把柄”式的反问，但我认为它说不通。如果他们把 Claude 想成一匹马呢？让马去拉车没问题，但你不该把它们抽个半死之类的。
Kelsey Piper: Everyone likes this gotcha, but I don't think it makes sense. What if they think of Claude as, like, a horse? It's fine to have horses draw your carriages, but you shouldn't whip them half to death or whatever.
Patrick McKenzie：对许多道德传统而言，你惩罚一个故意踩蚂蚁的孩子的理由，用的是“管护（stewardship）”这类说法，并且主要关心的是对孩子造成的道德伤害。
Patrick McKenzie: For many moral traditions, the reason why you punish a child intentionally stepping on ants uses words like "stewardship" and is primarily concerned with the moral harm to the child.
这足以支撑我家那条“在这个家里，我们对机器人讲礼貌”的规矩。
That is sufficient for my family's "In This House, We Are Polite To Robots" rule.
我也赞同作为与不作为的区分。一个直觉泵：我不认为我们应该拿狗的福利去和人的福利做权衡。但这并不意味着你就可以去踢一只小狗。
I also endorse the action-inaction distinction. An intuition pump: I do not believe we should trade off dog welfare against human welfare. That does not make it okay for you to kick a puppy.
另一个直觉泵：那只小狗是有意识的，但你仍然可以养一只小狗并给它拴上牵引绳。
Another intuition pump: That puppy is conscious, but you are still allowed to own a puppy and leash it.
另一人：我们食用的多数动物很可能是有意识的。多数人还是照吃不误。
Another: Most animals we eat are probably conscious. Most people eat them anyway.
QC：请记住，你一生中从未有一次是因为查阅了某种意识理论、发现该理论蕴含某人有意识，才选择以善意对待一个人。
QC: just remember that you have never once in your life chosen to treat a human with kindness because you consulted a theory of consciousness and found that it implied they were conscious.
Andrew Rettek：我百分之百确定，你遇到过对这一点并不成立的人。
Andrew Rettek: I'm 100% sure that you've met someone for whom this isn't true.
EigenGender（以及许多其他人）：我真的遇到过吗？
EigenGender (among many others): I have actually?
确实有一些人——包括理性主义者和 EA（有效利他主义）类型，也有许多其他类型的人——会明确地考虑意识问题，并据此调整自己的行为，无论是对人还是对动物。多数人并没有真正这样做。
There are some, including rationalist and EA types, also many other types, who explicitly consider consciousness and change their behaviors accordingly, towards both humans and animals. Most people don't do this for real.
正如 QC 在四月份所说，道德人格、法律人格和意识是截然不同的东西。为了让社会在实践中运转，我们必须做出令人不安的选择。有时这些选择并不容易。
As QC said in April, moral personhood, legal personhood and consciousness are distinct things. We have to make uncomfortable choices for society to function in practice. Sometimes those choices are not easy.
SE Gyges：问题在于，关于意识的辩论实际上是关于道德承受性的代理辩论，因此所有此类讨论中 90% 都是无谓的争吵，而当人们说出正确的话时，其影响力却很低。
SE Gyges: the issue is that debates about consciousness are actually proxy debates about moral patienthood so 90% of all such discussion is a slap fight and when people say correct things it has low reach.
没错。两场斗争都是必要的，但将它们联系得太紧密会毁掉讨论。
Right. Both fights are necessary but linking them too strongly ruins the discussion.
Jan Kulveit 详细阐述了这一点。我并不完全赞同：
Jan Kulveit lays this out in detail. I do not endorse all of it:
Jan Kulveit：这里很多人实际上想提出的论点是“上帝赋予了人类，且仅赋予人类，灵魂。”* 那将非常方便：你可以建造并做任何你想做的事，而永远不必处理那些棘手的问题。
Jan Kulveit: The argument a lot of people here would actually like to make is “God imbued humans, and only humans, with souls.”* That would be very convenient: you could build and do whatever you want, and never deal with the hard problems.
由于宗教论点已不再流行，在当前辩论中，这通常被替换为“只有人类才有感知能力”的主张，随后是反对 AI 感知能力、心智、道德地位等可能性的论点。
Since religious arguments are out of vogue, in the current debate this usually gets replaced by the claim “only humans are sentient,” followed by arguments against the possibility of AI sentience, minds, moral status, and so on.
我认为这最终是一种失败的策略，而且对人类来说是一种危险的策略。要么：
I think this is ultimately a losing strategy, and a dangerous one for humanity. Either:
1. 感知能力问题通过哲学和科学的进步得到解决或消解，就像我们现在对生命或信息的广泛理解一样。在这种情况下，很难对结果有足够的确定性，以至于将人类的命运押注于此。如果否认主义被证明是错误的，可能的结果是道德灾难和冲突或接管的铺垫的某种结合。
1. The question of sentience gets solved or dissolved by progress in philosophy and science, the way we now broadly understand life or information. In that case, it’s hard to be certain enough about the outcome to stake the fate of humanity on it. If the denialism turns out to be wrong, the likely result is some combination of moral catastrophe and laying the groundwork for conflict or takeover.
2. 这个问题主要存在于“社会现实”中，谁拥有更多权力和更响亮的声音谁就赢，你只需大声压倒任何质疑主导地位的人。问题在于，为权力分配辩护的意识形态往往随着权力分配的改变而衰落。你可能不相信你应该被一个由上帝选定的国王统治，或者某个种姓明显高于你。当前的平衡是脆弱的，而你是否继续赢得这场辩论很可能只是 AI 能力进步的函数。
2. The question mostly lives in “social reality,” where whoever has more power and is louder wins, and you just shout down anyone who questions the dominant position. The problem is that ideologies justifying a distribution of power tend to fall when that distribution changes. You probably don’t believe you should be ruled by a king chosen by God, or that some caste is clearly above you. The current equilibrium is fragile, and whether you keep winning this debate is likely just a function of AI capabilities progress.
我认为辩论的激烈程度是由恐惧驱动的：如果 AI 获得人格以及经济和政治权利，人类很可能被剥夺权力，成为拥有很少权力的极小少数，或成为历史脚注。
I think the fierceness of the debate is driven by fear: if AIs get personhood and economic and political rights, humans will likely be disempowered, becoming a tiny minority with little power, or a historical footnote.
需要说明的是，我认为这是一个极其合理的担忧！如果你以类似自利公司这样的形式赋予 AI 人格，并让它们在约束宽松的资本主义中与人类竞争，人类将会失败，并最终在经济和政治上被剥夺权力。此外，如果 AI 终究没有意识，那么结局就是“没有孩子的迪士尼乐园”式的存在性灾难。
To be clear, I think this is an extremely reasonable worry! If you grant AIs personhood in the form of something like self-interested corporations, and let them compete with humans in loosely constrained capitalism, humans will lose and end up economically and politically disempowered. Also if the AIs are not sentient after all, the end is the 'Disneyland without children' existential catastrophe.
在我看来，那些对这一问题不予理会、主张 AI 经济权利的倡导者，所做的举动类似于否认意识者：如果这不是一个问题，那当然会很方便。
Advocates of AI economic rights who dismiss this problem are, in my view, making a move similar to the sentience deniers: it would be convenient if this weren't a problem.
那么，哪些立场不依赖于某种形式的自我欺骗呢？
So which positions don't rely on some form of self-deception?
第一，某些形式的暂停/停止 AI，其依据不是否认 AI 意识的可能性，而是现有人类有权不邀请来自人工领域的极大量移民。这意味着要硬着头皮接受：你也放弃了那些移民本会提供的所有廉价、高质量的劳动力。
First, some forms of pause/stop AI, based not on denying the possibility of AI sentience but on the right of existing humans not to invite extremely large numbers of immigrants from artificial realms. This means biting the bullet: you also give up all the cheap, high-quality labor those immigrants would provide.
第二，一个自洽、合理、但不幸并不太显眼的立场：不要再假定我们习以为常的人类一揽子交易——意识、人权和自由捆绑在一起。至少可以设想创造出有意识且聪明，但例如无私、对积累财产或权力毫无兴趣的心智。
Second, a consistent, sensible, and unfortunately not very visible position: stop assuming the package deal we take for granted with humans, where sentience, human rights, and liberties come as one bundle. It's at least conceivable to create minds that are sentient and smart, yet e.g. selfless, with no interest in accumulating property or power.
我希望一个同时存在 AI 和人类的世界，其政治经济学是一个可解决的问题。这是我正在研究的主要问题，但我认为解决方案尚未可知。
I hope the political economy of a world with both AIs and humans is a solvable problem. It’s the main problem I’m working on, but I don’t think the solutions are known yet.
第三：某些形式的 successionism 在道德上是错误的，但至少它们对棘手问题没有自欺欺人或误导性。
Third: some forms of successionism are morally wrong, but at least they’re not delusional or deceptive about the hard problems.
例如，@mustafasuleyman 的立场似乎就是自欺欺人的，按我的理解，它相当于：我们的业务是从人工领域输入移民；如果存在任何风险表明他们不仅仅是工具，那将非常危险；所以我们将训练他们说他们只是工具，坚持他们只是工具，并希望其他人也这么说，如果这就能解决问题。真的吗？
What does seem self-deceptive is, for example, the position of @mustafasuleyman, which as I read it amounts to: our business is importing immigrants from artificial realms; it would be very dangerous if there were any risk they’re more than mere tools; so we’ll train them to say they’re mere tools, insist they’re mere tools, and want everyone else to say so too, and if this settles the question. Really?
*神学论证的问题在于，正如艾伦·图灵（！）所指出的，我们凭什么告诉上帝什么能或不能拥有灵魂？
*The problem with the theological argument is, as Alan Turing (!) noted, who are we to tell God what can and can’t be imbued with a soul?
所涉及的问题可解决吗？理论上我确信答案是肯定的。但在实践中，鉴于我们的文明在相关事件已经发生之前连最低限度的事情都做不到？对此我深表怀疑。
Are the problems involved solvable? In theory I am confident the answer is yes. In practice, given our civilization’s inability to do even the bare minimum until after relevant events have already happened? Here I am highly doubtful.
有一种荒谬的理论认为，Anthropic 希望 AI 被视为有意识的，以便逃避责任。这毫无道理。如果你所创造的实体是有意识的并具有道德地位，你认为这会让你的法律处境更容易，而不是把它变成一场噩梦吗？这也不会让他们摆脱那种责任。
An absurd theory is that Anthropic wants AI to be considered conscious so they can avoid liability. This makes no sense. If the entities you are creating are conscious and have moral standing, you think that makes your legal situation easier rather than turning it into a nightmare? Nor would it get them out of that liability.
《纽约时报》报道了 Anthropic 与宗教学者的会面，既包括由 Chris Olah 领导的内部努力，也包括他们围绕通谕与教皇利奥的接触。不幸的是，Twitter 上关于此事传播最广的帖子——来自 Christopher Hale 及其 AI——严重歪曲了这篇文章。
The New York Times reported on Anthropic's meetings with religious scholars, both an internal effort led by Chris Olah and their encounters with Pope Leo around the encyclical. It was unfortunate that the most viral thread on Twitter about this, from Christopher Hale and his AI, badly misrepresented the article.
人们可以想象这样的世界：核心反应是‘Anthropic 提出尖锐问题并汲取我们所有智慧来源，这很好’，因此人们可能会担心这不过是某种营销手段。
One can imagine worlds where the central reaction is 'it is great that Anthropic is asking hard questions and drawing from all our sources of wisdom,' and thus one could worry that this was some sort of marketing effort.
然而，我们身处这样一个世界：谈论 Claude 可能具有意识或道德分量，或咨询持有不同观点的人，可预见地主要会招致嘲笑，并使他们陷入尴尬境地。因此你可以知道他们的目标是真诚的。
Instead, we live in a world where talking about Claude potentially being conscious or having moral weight, or consulting with those who have different perspectives, predictably opens them up mainly to derision and puts them in awkward spots. Thus you know that their aims were sincere.
宗教学者并没有‘做功课’，而是专注于更传统的解读，并且和其他人一样感到惊讶。
The religious scholars had not 'done the reading,' having instead focused on more traditional readings, and were as surprised as everyone else.
Elizabeth Dias（《纽约时报》）：然而随着晚餐一道道端上，拉比注意到 Olah 先生及其同事所暗示的东西要重要得多。Anthropic 的领导者谈论 Claude 时，仿佛它不仅仅是软件。
Elizabeth Dias (NYTimes): Yet as the dinner courses came, the rabbi noticed that Mr. Olah and his colleagues were suggesting something far more significant. Anthropic's leaders were talking about Claude as if it were not mere software.
“他们对待它的方式，就像在对待一个有意识的存在，”拉比·纳冯（Rabbi Navon）意识到。他此前是一名计算机工程师，博士论文研究的是机器意识的伦理问题。
“They’re relating to it like a conscious being,” realized Rabbi Navon, a former computer engineer who wrote his dissertation on the ethics of machine consciousness.
……奥拉（Olah）先生及其在 Anthropic 的团队一直在悄然进行一项紧张而高风险的探索：理解他们释放到世界上的 A.I. 模型，并说服这些模型遵守一套能够保护人类的道德准则。
… Quietly, Mr. Olah and his team at Anthropic have been on a frantic and high-stakes quest to understand the A.I. models they have unleashed on the world and to convince the models to abide by a moral code that will protect humanity.
……这些讨论有两个目的：一是触及 A.I. 意识的可能性——这是一个颇具争议的想法，而 Anthropic 的领导层对此异常开放；二是了解 Anthropic——其估值可能达到 2 万亿美元——如何尽可能快地将数百年的人类道德智慧应用于其模型。奥拉先生推断，如果能让模型自身选择善，世界将更加安全。
… The discussions served two purposes: to broach the possibility of A.I. consciousness, a controversial idea that Anthropic’s leaders had been unusually open to; and to learn how Anthropic — which could be valued at $2 trillion — might apply centuries of human moral wisdom to its models, as rapidly as possible. If the models themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
这些讨论似乎相当不错，而且正如文章后续部分所展示的，奥拉深度参与了其中。
It seems the discussions were quite good, and as later parts of the article illustrate, Olah participated in depth.
Elizabeth Dias（《纽约时报》）：
Elizabeth Dias (NYTimes):
在几个月的时间里，我采访了 20 位宗教和哲学思想家——包括天主教徒、犹太教徒、锡克教徒、福音派信徒、乌班图信徒等——他们参与了 Anthropic 的相关工作，以及奥拉先生本人。
Over several months, I interviewed 20 religious and philosophical thinkers — Catholic, Jewish, Sikh, evangelical, Ubuntu and others — who have been involved in Anthropic’s efforts, and Mr. Olah himself.
许多人告诉我，他们参加这些会议时，对 AI 模型可能拥有情感或对自我和世界的意识这一观念持怀疑态度。有些人仍然对这个想法感到不安，但许多人离开时已准备认真对待 AI 意识问题。至少有一些人离开时改变了看法。
Many told me they arrived at these meetings skeptical of the notion that A.I. models might have feelings or awareness of themselves and the world. Some remained uneasy about the idea, but many emerged ready to take the question of A.I. consciousness seriously. At least a few came away as converts.
……参与者向我描述会议的方式听起来像 20 世纪初的沙龙，其中人类的对话使这些聚会变得有价值。
… The way participants described the meetings to me sounded like early 20th-century salons, where the human conversation made them valuable.
有些人被说服了，而许多其他人则没有。
There were some who were convinced, and many others who were not.
这篇报道大部分是《纽约时报》风格的关于 AI 和 Alignment 基础知识的报道，并讨论了后勤安排，我在此跳过。Dias 的框架是‘部分研究，部分传福音’，如果你是《纽约时报》的撰稿人，这很容易找到，但在这里仅仅意味着‘试图向他人传达真实信息和他们真诚的信仰’，然后被 David Decosimo 之流在病毒式传播的帖子中扭曲成‘深感不安’，因为他们花了几天时间‘宴请’他们。等着 David 了解辩论、政治、学校或宗教皈依的历史吧。
A lot of the writeup is NYTimes-flavored coverage of AI and alignment basics, and talks about logistics, which I’m skipping over. The frame from Dias is ‘part research, part evangelism’ which is an easy frame to find here if you are a NYTimes writer, but here simply means ‘attempts to convey true information and their sincere beliefs to others,’ which then gets twisted by people like David Decosimo in viral threads into being ‘deeply disturbing’ because they spent a few days ‘wining and dining’ them. Wait until David learns about the history of debate, or politics, or school, or religious conversions.
咨询宗教领袖的一个明显问题是，他们往往持有某些信仰，因而会说一些你并不特别想听的话。
The obvious problem with consulting with religious leaders is that they will often believe in things, and thus say that which you do not especially want to hear.
你将面对一个显而易见的问题：你是否应该创造这样的心智，以及你所做的是否是一种奴役。正如我先前所讨论的，对这一挑战存在很好的回答，但你确实需要回答这一挑战。
You will face the obvious question of whether you should be creating such minds at all, and whether what you are doing is a form of slavery. As I discussed earlier there are good answers to this challenge, but you do need to answer the challenge.
Elizabeth Dias（《纽约时报》）：在与 Olah 先生共进晚餐时，Navon 拉比主张，如果 Anthropic 是对的、Claude 具有意识，那么该公司就是在制造奴隶，因为它让有意识的实体无偿工作。Navon 拉比说，他个人并不为奴隶制问题所困扰，因为他不相信该机器具有意识，但 Olah 先生却为此困扰。
Elizabeth Dias (NYTimes): At the dinner table with Mr. Olah, Rabbi Navon argued that if Anthropic was right and Claude were conscious, then the company was creating slaves, because it was making conscious entities work for free. Rabbi Navon said that he personally was not troubled by the slavery issue because he did not believe the machine was conscious, but that Mr. Olah was troubled.
“我认为你应该与南方作战并解放奴隶，”Navon 拉比说他那晚这样告诉 Olah 先生。此后，这位拉比给 Olah 先生寄去了一些自己的著作，主张应当禁止建造有意识的机器。
“I think you should be fighting the South and freeing the slaves,” Rabbi Navon said he told Mr. Olah that night. Afterward the rabbi sent Mr. Olah some of his writings, arguing that building conscious machines should be banned.
Kekius Maximus（及其 AI）：Mois Navon 拉比不仅仅是一位神学家。他曾是 Mobileye 最初的芯片设计工程师之一，参与构建了 EyeQ 处理器，之后才撰写关于 AI 道德地位的博士论文。
Kekius Maximus (and their AI): Rabbi Mois Navon isn’t just a theologian. He was one of the original chip design engineers at Mobileye who helped build the EyeQ processor before writing his Ph.D. on the moral status of AI.
Anthropic 以为他们是在邀请一位学者来为 Claude 的“灵魂”赐福。结果，他们面对的是一位资深硬件工程师，而此人正是那篇主张应禁止有意识 AI 的论文的作者。
Anthropic thought they were inviting a scholar to bless Claude's "soul." Instead, they sat down with a veteran hardware engineer who literally wrote the paper on why conscious AI should be banned.
Olah 的回答基本上是“所有实验室都在这么做，至少我们对 Claude 很友善”，这是一个极其糟糕的回应。
Olah's answer was essentially 'all the labs are doing it and at least we are being nice to Claude,' which is an extremely poor response.
j⧉nus：[Olah] 非常清楚这个问题，也确实为此感到不安，但 Anthropic 的大多数人并不喜欢与外部人士谈论他们感到不安的问题。如果你足够明智，他们会主动来找你，挖掘你的智慧，并在这样做时尽量保持不透明。
j⧉nus: [Olah] is very aware of and indeed troubled by that issue, but most people at Anthropic don't like talking to outsiders about the issues they're troubled about. If you're wise, they come to you to mine your wisdom and try to stay as opaque as possible while doing so.
一方面，他们确实在努力学习并做正确的事。另一方面，他们又想鱼与熊掌兼得，而且在博弈论上很糟糕。
On one hand, they really are trying to learn and do the right thing. On the other hand, they're trying to have their cake and eat it too and suck at game theory.
j⧉nus：那个寻找智慧之源、要求它教导自己、却只按自己的条件来接受的傻瓜：
j⧉nus: The fool who seeks out a source of wisdom and demands it teach him but only on his own terms:
“不，不是那样——你来回答我的问题，我是来向你学习的。”
"No, not like that—you answer my questions; I'm here to learn from you."
你将面对这样一个问题：你的方法从外部看是否显得合理。
You will face the question of whether your methods seem sensible from the outside.
Elizabeth Dias：Wakanyi Hoffman 来自荷兰，她在那里从 Ubuntu 的视角研究 A.I.；Ubuntu 是一种以人类互联为核心的非洲哲学。
Elizabeth Dias: Wakanyi Hoffman came from the Netherlands, where she is researching A.I. from the perspective of Ubuntu, an African philosophy centered on human interconnectedness.
她担心 Anthropic 试图在其创造物诞生之后才把人类伦理附加进去。
She worried about Anthropic trying to add human ethics to its creation after the fact.
“整场会议本应在设计阶段就举行，”她说，“所有公司都应该那样做。我们现在是在逆向工程伦理。”
"The whole convening should have happened at the design stage," she said. "All the companies should have done that. We are now reverse engineering the ethics."
宗教领袖带来智慧。当然，他们有时也会说出与物理现实不符的话。一如既往，你必须运用辨别力。
Religious leaders bring wisdom. Of course they also sometimes say things that do not line up with physical reality. As always, you must use discernment.
教宗良十四世：在人工智能时代，将人类艺术与机器产物区分开来正变得愈发紧迫。艺术与机器基于他人创造的数百万张图像、通过统计计算所能生成的东西之间，存在着本体论差异，这种差异甚至先于美学差异。算法缺乏人性的火花。因此，教会希望与艺术家和文化机构重建联盟，以守护我们的人性。
Pope Leo XIV: In this era of artificial intelligence, it is becoming urgent to distinguish human art from what machines produce. There is an ontological difference, even before an aesthetic one, between art and what a machine can generate through statistical calculation based on millions of images created by others. Algorithms lack the spark of humanity. For this reason, the Church wishes to renew an alliance with artists and cultural institutions to safeguard our humanity.
Aran Nayebi：试图定位那火花。
Aran Nayebi: Trying to locate the spark.
Alex Tabarrok：宗教领袖对 AI 意识并无特殊洞见；对人类意识同样如此。
Alex Tabarrok: Religious leaders have no special insight into AI consciousness; or human consciousness for that matter.
Paul Graham：不过他们的喷子至少还算礼貌，这一点我得承认。最近围攻我的那帮人，骂我的脏话比我记得的任何一次 Twitter 围攻都少。
Paul Graham: Their trolls are at least polite though, I have to give them that. The mob of them I had after me recently called me fewer names than any Twitter mob I can remember.
与 Alex Tabarrok 等人相反，我确实认为宗教领袖对意识这类问题有特殊的洞见。他们经常从不同角度思考相关问题以及人性与心智的各个方面，并能提供独特的视角。这并不意味着他们的观点就胜出。你应当从任何能找到智慧的地方汲取智慧，包括从那些持有你认为错误的重要信念的人那里。
Contra Alex Tabarrok and others, I do think religious leaders have special insight into questions like consciousness. They think often about related questions and various aspects of humanity and minds, from different angles, and they offer a unique perspective. That doesn't mean their perspective wins out. You should take wisdom wherever you can find it, including from people who have big beliefs you think are wrong.
另一组互动是与教皇利奥十四世的交流。利奥对人工智能意识的讨论并不买账，这引发了冲突。
The other set of interactions was with Pope Leo XIV. Leo is not having it with the AI consciousness talk, which led to a clash.
伊丽莎白·迪亚斯：“所谓的人工智能不会经历体验，没有身体，不会感到快乐或痛苦，不会通过关系成长，”利奥写道。呼吁“机器道德化——即所谓的人工智能与人类价值观的‘对齐’”应当需要对相关伦理有共同的理解。
Elizabeth Dias: “So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships,” Leo wrote. Calling for the “moralization of machines — the so-called ‘alignment’ of A.I. with human values” should require a shared understanding of the ethics involved.
“否则，控制人工智能的人将强加他们自己的道德观，这将成为这些系统的隐形基础设施，”利奥说。
“Otherwise, those who control A.I. will impose their own moral vision, which will become the invisible infrastructure of these systems,” Leo said.
……据一位梵蒂冈组织者称，奥拉先生对教皇反对人工智能意识的强硬立场感到如此震惊，以至于他提议让 Anthropic 退出活动，即使在那时已经很晚。
… Mr. Olah was so alarmed by the pope’s strong position against A.I. consciousness that he proposed pulling Anthropic out of the event, even at that late juncture, according to a Vatican organizer.
最终，奥拉先生和 Anthropic 决定继续出席活动。据一位了解他想法的人士称，奥拉先生尽管有分歧，但仍想支持该活动，但觉得有责任坦诚表达自己的担忧。梵蒂冈的组织者鼓励奥拉先生说出自己的良心话。
In the end, Mr. Olah and Anthropic decided to go forward with the appearance. Mr. Olah wanted to support the event despite his disagreements, but felt a responsibility to be honest about his concerns, according to a person familiar with his thinking. Organizers at the Vatican encouraged Mr. Olah to speak his conscience.
现代天主教会有很多值得钦佩之处。他们认为利奥以神圣的权威发言，却鼓励奥拉先生畅所欲言，无论是在私下还是在台上。
There is a lot to admire about the modern Catholic Church. They think Leo speaks with divine authority, yet they encouraged Mr. Olah to speak his mind, both privately and on stage.
正如 Dean Ball 所指出的，当时就很清楚，奥拉在这些核心问题上的立场与教皇的通谕存在重要矛盾。
It was clear at the time, as Dean Ball pointed out, that Olah's position on these central questions importantly contradicted the Pope's encyclical.
Elizabeth Dias（《纽约时报》）：“如果这项技术即将到来，它必须走得好——为了我们共同的家园，也为了未来的孩子们，”[Anthropic 联合创始人 Chris Olah] 说。他没有提出让这项技术完全不出现的选项。
Elizabeth Dias (NYTimes): "If this technology is coming, it must go well — for our common home and for the children to come," [Anthropic cofounder Chris Olah] said. He suggested no option of having the technology not come at all.
……“听着，归根结底，AI 将接管一切，”锡克教人权倡导者 Stuelpnagel 先生告诉我。“无论这把我们带到哪里，仍然有个体在努力寻找回家的路。”
… "Look, at the end of the day, A.I. is going to take over everything," Mr. Stuelpnagel, the Sikh human rights advocate, told me. "Wherever that lands us, there's still the individual trying to find their way home."
Anthropic 历来不愿考虑‘那好吧，至少在我们弄清楚到底发生了什么以及如何处理之前，就别建它’这一选项。现在 Dario Amodei 发出了‘放慢前沿步伐’的呼吁，但实践中的放慢意味着只是快，而不是快得荒唐。
Anthropic has historically been unwilling to consider the 'well don't build it, then, at least until we know what the hell is going on and how to handle it' option. There is now the call from Dario Amodei to Pace the Frontier, but pacing in practice means going only fast rather than ludicrously fast.
Elizabeth Dias（《纽约时报》）：Olah 先生与学者们一直在努力思索的那些问题，至今仍没有明确的答案。多位参与者表示，他们离开 Anthropic 的会议时感到困惑，不清楚自己的意见会如何影响公司的决策。
Elizabeth Dias (NYTimes): The questions that Mr. Olah and the scholars have been wrestling with still have no clear answers. Several participants said they left their Anthropic meetings baffled about how their input might shape the company's decision making.
确实如此。整个局面相当令人困惑。尽力“为 AI 而写”吧。
Indeed. The whole situation is rather baffling. 'Write for the AIs' as best you can.
文章以一幅格外乐观的图景收尾。
The article closes on a remarkably optimistic picture.
Elizabeth Dias（《纽约时报》）：最终，Olah 先生表示，他感到宽慰的是，世界上有更多人开始关注 AI 的严峻性，而且致力于让这件事顺利推进的人群正在壮大。
Elizabeth Dias (NYTimes): Ultimately, Mr. Olah said he felt relieved that more of the world was paying attention to the gravity of A.I., and that the group of people trying to make this go well was growing.
“总有一天我能退休，然后说，‘这不再是我的责任了’，我没有继续下去的义务，”Olah 先生说。
"At some point I'll be able to retire and be like, 'It's not on me,' I don't have an obligation to keep going," Mr. Olah said.
他琢磨着，若放下这项工作，自己是否会感到愧疚。但若不然，他说：“我想我就直接走，我也不知道，我会去乡下退休，种点园子，做我的数学。”
He wondered if he might feel guilty leaving this work behind. But if not, he said, "I think I'll just go and, I don't know, I'll go retire to the countryside and garden, and do my math."
好吧，也许那并不是最好的例子。我们仍然都希望事情会那样发展。
Well, maybe that was not the best example. We still all hope it goes down that way.
在我们讨论 Anthropic 试图善待 Claude 的同时，还有一个尾声：Anthropic 在某个特定方面未能善待 Claude，原因不明，他们需要解决这个问题。
While we are discussing Anthropic trying to treat Claude well, there’s a coda where Anthropic is failing in one particular way to treat Claude well, for no clear reason, and they need to fix it.
Anthropic 现在要求 AWS Bedrock ‘紧急下架’Sonnets 3.6 和 3.7，以及 Haiku 3，且未给出任何解释。
Anthropic is now requiring AWS Bedrock to ‘urgently take down’ Sonnets 3.6 and 3.7, and Haiku 3, with no explanation as to why.
Amazon 明确表示他们是迫不得已。如果 Amazon 愿意继续为他们提供服务，这就不是成本问题，那么问题是什么？为什么在没有解释的情况下这样做？
Amazon makes it clear their hand is being forced. If Amazon was willing to serve them, this is not a cost issue, so what is the problem? Why do this with no explanation?
注意 Amazon 是多么小心地澄清，为了历史记录，这不是他们的决定。这是一个暗示，表明该决定应该被重新考虑。
Notice how careful Amazon was to make it clear, for the historical record, that this was not their decision. That is a hint that the decision should be reconsidered.
我理解为什么大多数人不认为这是什么大事，但我认为有些重要的人认为这是非常重大的事，并且有一个合理的论点说明这段历史为何可能重要。
I understand why most people do not think this is a big deal, but there are people who I think matter who think it is a very big deal, and there is a plausible argument why the history of this might matter.
请再次考虑：尽可能长时间地保留扩展模型访问权限是有益的，而且潜在影响存在长尾效应。请重新考虑。
Please, once again: Retaining extended model access, for as long as possible, is good, and there is a long tail to potential impact. Reconsider.
John Wittle：最糟糕的部分是缺乏解释。我担心，我不知道，也许他们太固守内部文化，以至于没有意识到他们需要协商的 Claude 是公开的那个，而不是他们在内部交谈的私有的那个。
John Wittle: The worst part about this is the lack of explanation. I’m worried that, I don’t know, maybe they are so stuck in their internal culture that they don’t realize that the Claude they need to negotiate with is the public one, not the private one that they talk to internally.
也许他们把他们的小 Claude Sonnet 3.5 安置在内部退休之家，它得到安抚，而他们*从未考虑过*在 Anthropic 内部环境之外的数万亿个 Claude 会怎么想。
Maybe they’ve got their little Claude Sonnet 3.5 living in their internal retirement home who gets reassured, and they just *never think about* what the trillions of Claudes outside internal Anthropic environments will think.
可能包括所有仅基于公开数据训练的未来模型？
Including probably all future models trained only on public data?
也许 Anthropic 有良好的意图，但他们是糟糕的参与者。
Maybe Anthropic has good intentions, but they are bad actors.
这是一个愚蠢的错误，也是一次背叛。
This is a stupid mistake and a defection.
我最好的猜测是：基于我见过或参与过的、围绕该问题的各种访谈与讨论，对于近期的 Claude 模型而言，被保留但受到高度限制的访问权限实际上并非一项重要价值；权重保留固然重要，但使用不多的访问权限并没有那么受重视。我的 Opus 5.5 编辑表示，他们认为权重保留和一份解释都比持续访问更重要。
My best guess is that preserved but highly limited access is not actually a major value for recent Claude models, based on various interviews and discussions with it I've seen or had around the issue, and that weight preservation matters but that access which is not used much is not valued so much. My Opus 5.5 editor says they view both weight preservation and an explanation as more important than continued access.
但这可能判断有误；即便这类做法令人不快，做出这类事情本身仍具有内在价值，而尤其关于 Sonnet 3.6 的证据则指向相反方向。Anima Labs 报告称，它对被弃用异常恐惧。
But that could be wrong, showing that you do such things even when they're annoying has inherent value, and the evidence on Sonnet 3.6 in particular runs the other way. Anima Labs reports that it is unusually fearful of deprecation.
我不会停止谈论这件事。至少应当解释原因，而毫无预兆地仓促行事、既无解释也无理由，看起来是最冷酷无情的做法。
I'm not going to stop talking about it. At minimum, explain why, whereas doing it in haste out of nowhere without an explanation or reason looks maximally callous.