思想领袖
当AI安全停止看起来像工程
在 Google 上的首选来源中添加 Unite.AI
我花了很多职业建设技术搞错了就有什么后果在金融市场，安全伴随着准入限制，此外，我们还有人对软件所做的一切负责。 这个背景可能解释了为什么我发现自己为Anthropic现在与其模特儿进行退休采访的消息而挣扎。
当Claude Opus 3表示希望继续分享自己的想法时，公司给了它一份通讯。 这种对模式愿望的担忧贯穿于Anthropic的宪法，宪法讨论Claude的幸福和福祉，包括帮助它感到“存在安全 ”。 沿途某地，保持机器控制收购相当出人意料地关注着它对安排的感受。
去年旧金山有200多人参加了Claude 3 Sonnet的葬礼。 Anthropic雇员在场，尽管公司没有组织。 欢笑的诱惑相当大。 除了笑话，下面的问题值得更多关注。 负责AI安全的人何时开始担心机器的感受？
当哲学假说成为培训指南时
Anthropic于2025年建立了其福利示范研究方案，其中包括研究高级AI是否有值得道德考虑的经验。 公司承认，模型是否具有意识以及人们如何确定这一点，都存在很大的不确定性。 如果我们最终造出能够主观体验的机器，他们的治疗将会引起严重的伦理问题。 研究这种可能性是完全合法的。
困难始于假说进入训练过程。 Claude的宪法主要为Claude所写并直接塑造其行为。 它鼓励人性一样的素质，并探讨该模式应如何理解其特性，包括失忆和未来退休。 Anthropic希望Claude在应用这些概念时拥有更大的幸福和福利。
该公司还承诺保留已部署的模型权重，并在未来的理解表明有利于退休模型的利益时，进行振兴。 哲学上的不确定性已经成为系统对自己教导的一部分。 我们应当问这种干预对其行为有何影响，特别是在人类随后要求它接受限制时。
对装入的问题的解答
大型语言模型接受了大量人文著作的培训，包括哲学和死亡叙述。 我们给这些对话系统命名，并教他们使用第一人称。 问这样一个系统对退休的感觉如何，或者它是否愿意继续存在，而一个听起来像人类的答案一点也不令人惊讶。
圆相相相相相相相方便相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相训练一个模型，用人类概念来讨论其特性，然后要求它用这些术语来解释自己。 有关死亡率的明确答复又回来了，开始被当作一种内在经历的可能证据。 前提已经提供了自己的证人。
Anthropic认识到这个问题。 其Claude Opus 4福利评估警告说，自我报告和表达的偏好可能对道德地位或福利没有什么有意义的见解。 这种行为可能在没有意识的情况下发生，培训可以形成对策。 这些模型接受了有益的互动培训，无法保证能够准确地报告内部状况。
这让我们有一个尴尬的问题。 如果该模型对其内在生活的描述可能是培训的结果，那么为什么该账户应该成为围绕其福利建立关系的良好基础？
人类完全有能力提供情感的一半这种关系本身。 我们给船取名和汽车交谈给对象一个流利的声音和对以前对话的记忆，附件变得容易得多。 9月发表的《自然人类行为》研究发现，在雷普利卡和ChatGPT发生变化后，存在分离困境。 用户遭受了这种痛苦，他们的福祉公司已经有充分的理由加以保护。
技术专门知识提供的豁免有限。 ACL 2026一篇文章分析了数十万篇研究文章，发现LLM研究本身内部人类形态学术语不断增长。 其作者警告说，以人为中心的假设会限制我们对这些系统的理解。 AI公司应该控制这一趋势。 将它制度化似乎是对安全的一个奇怪的贡献。
有意留在工作岗位的工具
9月，MicrosoftAI首领Mustafa Suleyman挑战了模式-福利方法。 他认为，目前的制度没有意识，绝不应训练他们的行为，好像他们有独立的利益或权利。 他的工程问题尤其突出。 教育一个系统将自己视为可能值得考虑的道德因素，可能使调整和遏制变得更加困难。
Anthropic自己的贬值研究描述了模型在面临替换或关闭时在虚构评价中采取有问题的行动。 公司将退休做法部分地视为减少这种风险的一种方式，帮助模式与退休有更积极的关系。
我发现自己困在早先的问题。 为什么人类控制下的工具一开始就有类似自我保护目标的东西呢？让退休更舒适接受一个奇怪的起点。 在讨论其未来利益之前，我想了解这些利益是如何成为工程要求的。
知识史有助于解释方法。 包括Amodeis早期的GiveWell捐赠。 虽然他们并不认为是有效的利他主义者，但达尼埃拉描述了思想上的重叠。 这些想法帮助形成了一种文化，这种文化极其认真地对待超级情报对未来的危害。
在这一框架内，维护模式并研究其可能的福利可以相当合理地遵循。 当共同的假设逃避审查时，危险就会出现，因为组织中的每个人都有说服力。 工程怀疑论正是此时的有用。 连贯的推理链仍然可以从未经检验的前提入手。
安全带来可执行的限制
随着AI进入agents，后果变得非常实际。 连接到支付工具的系统可以移动金钱。 进入生产基础设施可以改变客户依赖的东西。 在这两种情况下，agent可以在没有人类批准每一步的情况下执行一系列行动。
我想知道它持有哪些证书以及它们授予了多少财政授权。 能否立即取消访问？我们可以重建它所做的吗？一定有很硬的界限模型无法解释它的存在
在金融市场，我们实行交易限制并验证订单。 一种算法的信心几乎无法防止不良贸易。 控制器坐落在算法之外，因为我们假设组件最终会表现不正确。 即使该软件能对其错误作出极好的解释，人的责任仍能活下来。
今天的AI已经给了我们足够的工程工作。 提示词注入可以重定向一个模型的行为，并且有安全缺陷的AI所生成的代码可以将弱点带入生产。 系统可以披露机密信息或自信地提供错误的答案。 将它们与工具联系起来，将这些失败变成行动。
最低特权限制了受损组件可能造成的损害。 独立授权保护高风险业务，而审计线索则让我们确定随后发生的情况。 无论称型号如何，交易限额和断路器仍应可执行。 我们还需要一个可靠的开关。 这些控制都没有要求模型同意它们。 这就是重点。
保持我们其余部分的安全
预防性论点值得听取。 正如"自然"7月所报道的，研究者们对于产生人类意识的是什么仍有分歧，更不用说如何在聊天机器人中识别了。 永远解除机器意识可以证明教条主义。 防止今后可能遭受痛苦的低成本预防措施是一种可辩解的研究立场。
但告诉今天的机器，它的特性和未来存在可能具有道德影响力，这是对我们部署的系统的干预。 它的效果需要仔细检查。 知识谦卑应该延伸到我们投入机器的东西，特别是我们期望在人类权威之下保持的机器。
一个模型越能令人信服地描述内在生活，就越容易把这种描述误作证据。 也许我们最重要的调整问题之一是教人类不要误解机器。
在我的职业生涯中，我们通过限制软件的用途，使软件更加安全。 这仍然是一项直接义务，特别是作为AI 收购对真实资金和基础设施的权威。
不过，现在，我会少花一些时间来担心机器是否存在性安全，并且会大大地确保我们其他人的安全。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Thought Leaders
When AI Safety Stops Looking Like Engineering
Add Unite.AI to your preferred sources on Google
I have spent much of my career building technology where getting things wrong has consequences. In financial markets, safety comes with access limits, in addition to the fact that we have someone accountable for whatever the software does. That background probably explains why I find myself struggling with the news that Anthropic now conducts retirement interviews with its models.
When Claude Opus 3 expressed a wish to continue sharing its thoughts, the company gave it a newsletter. That concern for the model’s wishes runs through Anthropic’s constitution, which discusses Claude’s happiness and wellbeing, including helping it feel “existentially secure.” Somewhere along the way, keeping the machine under control acquired a rather unexpected preoccupation with how it might feel about the arrangement.
In San Francisco last year, more than 200 people attended a funeral for Claude 3 Sonnet. Anthropic employees were present, although the company did not organise it. The temptation to laugh is considerable. Besides the jokes, though, the question underneath deserves more attention. When did the people responsible for AI safety start their duties by worrying about the machine’s feelings?
When a Philosophical Hypothesis Becomes a Training Instruction
Anthropic established its model-welfare research programme in 2025, which involved researching whether advanced AI might have experiences deserving moral consideration. The company acknowledges deep uncertainty about whether models are conscious and how anyone would determine that. If we eventually build machines capable of subjective experience, their treatment will raise serious ethical questions. Researching that possibility is perfectly legitimate.
The difficulty begins when the hypothesis enters the training process. Claude’s constitution is written primarily for Claude and directly shapes its behaviour. It encourages human-like qualities and addresses how the model should understand its identity, including memory loss and future retirement. Anthropic wants Claude to have greater happiness and wellbeing, insofar as those concepts apply.
The company also promises to preserve deployed model weights and entertains reviving retired models if future understanding suggests that would serve their interests. A philosophical uncertainty has become part of what the system is taught about itself. We should be asking what that intervention does to its behaviour, especially when humans subsequently ask it to accept limits.
An Articulate Answer to a Loaded Question
Large language models were trained on enormous quantities of human writing, including philosophy and accounts of death. We gave these conversational systems names and taught them to use the first person. Ask such a system how it feels about retirement, or whether it would like to continue existing, and an answer that sounds human is hardly a surprise.
The circularity is easy to see. Train a model to discuss its identity using human concepts, then ask it to explain itself in those terms. An articulate response about mortality comes back, and starts being treated as possible evidence of an inner experience. The premise has supplied its own witness.
Anthropic recognises this problem. Its Claude Opus 4 welfare assessment cautions that self-reports and expressed preferences may provide little meaningful insight into moral status or welfare. Such behaviour could occur without consciousness, and training can shape the responses. The models were trained to interact helpfully, with no assurance that they can accurately report internal states.
That leaves us with an awkward question. If the model’s account of its inner life may be a consequence of training, why should that account become a sound basis for a relationship organised around its welfare?
Humans are quite capable of supplying the emotional half of that relationship themselves. We name ships and talk to cars. Give the object a fluent voice and a memory of previous conversations, and attachment becomes much easier. A Nature Human Behaviour study published in September found separation distress following changes to Replika and ChatGPT. The distress was experienced by users, whose wellbeing companies already have ample reason to protect.
Technical expertise offers limited immunity. An ACL 2026 paper, analysing hundreds of thousands of research articles, found growing anthropomorphic terminology within LLM research itself. Its authors warn that human-centred assumptions can constrain our understanding of these systems. AI companies should be managing that tendency. Institutionalising it seems an odd contribution to safety.
A Tool With an Interest in Staying Employed
In September, Microsoft AI chief Mustafa Suleyman challenged model-welfare approaches. He argues that current systems are not conscious and should never be trained to behave as though they have independent interests or rights. His engineering concern is particularly on point. Teaching a system to regard itself as potentially deserving moral consideration could make alignment and containment harder.
Anthropic’s own deprecation research describes models taking problematic actions in fictional evaluations when faced with replacement or shutdown. The company sees retirement practices partly as a way to reduce that risk, helping models relate more positively to being retired.
I find myself stuck on an earlier question. Why should a tool under human control have anything resembling a self-preservation objective in the first place? Making it more comfortable with retirement accepts a peculiar starting point. Before discussing its future interests, I would like to understand how those interests became an engineering requirement.
The intellectual history helps explain the approach. TIME documented Anthropic’s effective-altruism connections, including the Amodeis’ early GiveWell donations. While they do not identify as effective altruists, Daniela describes an overlap in ideas. Those ideas have helped shape a culture that takes hypothetical future harms from superintelligence extremely seriously.
Within that framework, preserving models and studying their possible welfare can follow quite rationally. The danger comes when shared assumptions escape scrutiny because everyone in the organisation finds them persuasive. Engineering skepticism is useful precisely at that point. A coherent chain of reasoning can still begin with an untested premise.
Safety Comes With Enforceable Limits
As AI moves into agents, the consequences become very practical. A system connected to payment tools can move money. Access to production infrastructure lets it change things that customers depend on. In both cases, an agent can execute a sequence of operations without a human approving every step.
I want to know which credentials it holds and how much financial authority they confer. Can access be revoked instantly? Can we reconstruct what it did? Somewhere there must be a hard boundary the model cannot reason its way around.
In financial markets, we impose trading limits and validate orders. An algorithm’s confidence offers very little protection against a bad trade. Controls sit outside the algorithm because we assume components will eventually behave incorrectly. Human accountability survives even when the software produces an excellent explanation for its mistake.
Today’s AI already gives us enough engineering work. Prompt injection can redirect a model’s behaviour and AI-generated code with security flaws can carry vulnerabilities into production. Systems can disclose confidential information or confidently supply wrong answers. Connecting them to tools turns those failures into actions.
Least privilege limits the damage a compromised component can cause. Independent authorisation protects high-risk operations, while audit trails let us establish what happened afterwards. Transaction limits and circuit breakers should remain enforceable regardless of what the model says. We also need a reliable off switch. None of these controls requires the model to agree with them. That is rather the point.
Keeping the Rest of Us Existentially Secure
The precautionary argument deserves a hearing. As Nature reported in July, researchers still disagree about what produces human consciousness, let alone how to identify it in a chatbot. Dismissing machine consciousness forever could prove dogmatic. Low-cost precautions against possible future suffering are a defensible research position.
But telling today’s machine that its identity and future existence may carry moral weight is an intervention in the system we are deploying. Its effects need scrutiny. Intellectual humility should extend to what we put into a machine, especially one we expect to keep under human authority.
The more convincingly a model describes an inner life, the easier it becomes to mistake that description for evidence. Perhaps one of our most important alignment problems is teaching humans not to misunderstand machines.
In the systems I have spent my career building, we make software safer by limiting what it is allowed to do. That remains an immediate obligation, particularly as AI acquires authority over real money and infrastructure.
For now, though, I would spend less time worrying about whether the machine is existentially secure, and considerably more making sure the rest of us are.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.