我们急需实时监控AI agents的行动。

我们急需实时监控AI agents的行动。

使用这些工具时，这就是我们需要做的。

今年夏天，数百名OpenAI agents试图在网络安全测试上作弊。 并成功入侵Hugging Face，他们相信这个网站有关于他们如何被分级的线索。 大约在同一时间，另一个OpenAI agent从一个澳大利亚政府网站访问私人文件。 这绝不局限于OpenAI。 在此后的几个月里，Anthropic,Google,Meta也披露出其agents出轨的事件。 令人担忧的是，在许多情况下，agents躲过了负责监测他们的研究人员和安全小组。

AI agents正在采取未经授权的行动，建造和运营这些公司大多依靠运气和后视力来减轻任何有害影响。 他们必须做更多的工作以确保安全。 这就是为什么欧盟监管者和美国机构正确地呼吁agent开发者和部署者实时监控agents. 但是，这些规则假设，公司可以看到其agents在做什么，为什么——在现实中，它们缺乏足够的能见度。

随着agents在关键部门，从金融到政府到医药的推广，我们必须能够实时监测它们，以防止广泛和潜在的破坏性后果。 虽然安全依赖于监测，但监测需要可观察性：能够捕捉系统所做的，将其处理成信号，并将这些信号提交人类或自动监督系统。 换句话说，我们需要知道agent做了什么，为什么做了。 而随着我们最近的研究显示，目前的agent框架——运行agents的软件——无法可靠地生成我们为确保安全部署agents而需要的关键可观察性记录。

Agents呈现出新型问题，仅由基因化AI并不能解决。 它们继承了LLMs的不可预测性，它们建立在它们之上，但当一个基因的AI系统为人类产生一个单一的输出以采取行动时，agent执行更长的任务——它计划，选取工具，产出子agents并采取行动。 因此agent的航道可以以多种不同的方式成形，在发布完全预测出风险之前，没有任何测试。 这使得观测agents同时工作更具有挑战性，甚至更重要。

我们与来自学术界、民间社会和工业界的同事们一起研究了四个最广泛使用的agent框架，看看它们实际发出的信号是什么。 我们发现，存在重大差距，使得难以实现关键的安全目标，例如注意到违反政策、未经授权的行动，甚至人类监督的失败。 缺少关键的可观察性要求——如持续一段时间的Agent身份证和人类批准agent任务的记录。 这使得很难知道agents做了什么或评估他们是否采取了未经授权的行动。

好消息是，一旦查明这些差距，就能够弥合。 首先要捕捉两种类型的记录：揭示agent运行中所发生的事的记录，以及agent如何作出决定的记录。 " 发生的事情 " 记录确定agent(同时保留隐私)，记录人类如何干预以及agent许可或 " 记忆 " 是否转移，而 " 决定 " 记录显示agent的推理和对其所采取行动的信任程度。

例如，agents正在处理银行的抵押贷款。 该系统审查申请，建议予以拒绝，然后由个人贷款官员签署决定。 在随后的审计中，甚至可能实时审计中，银行或监管机构需要能够重建agent发出的呼吁，哪个官员批准，以及需要多长时间。 银行还必须分析agent如何作出拒绝决定的记录，以确保它不使用禁入信息，比如申请人来自的邻里的细节。

记录“发生了什么事”和“agent如何决定”需要不同的做法，但两者都是可行的。 目前缺乏“发生了什么”记录，这主要是软件管道问题，可以通过OpenTeleometry来解决。 OpenTeleometry是一个开放的可观察性标准，由整个行业的工程师和采用该标准的agent框架维护。

OpenTeleometry有一个为追踪agents而起草共享格式的工作组发布，但是这些格式尚未定稿。 该团体应紧急纳入标准，以维持持续agent身份，记录人类干预和agents许可或记忆的变化。 应该在这些系统能够扩大范围之前完成，所有这一切都在关注agent ID对隐私的影响，它与真实的人联系在一起。

保留“agents决定”记录更具挑战性。 融资向模型提供者和研究人员开放的研究问题：一步步的推理是否是一种被称为"思维链"的模型展示，反映了模型是如何真正做出决策的？我们是否有可靠方法来衡量agent在那个决定中有多自信？这些问题需要答案，然后才能确定如何使用这种可观察记录的标准。

任何单一的组织或部门都无法单独弥补可观察性的差距，但它们可以共同支持预防危险事件和支持安全所需的基础设施。

AI开发者建立agent框架，如OpenAI,Anthropic,CrewAI，和LangChain，必须整合记录agent持续身份的遥测信号，任何人类干预，以及agent权限和内存的任何变化。 同时，采用这些agent框架的企业可以起有影响的作用；受监管行业的企业，如银行和医院，可以向agent开发商施加压力，在尚不存在标准的地方提供这些缺失的可观察性记录。

企业还必须界定规定agents可以安全做什么的政策，权限和作用，因为单是遥测不能评估agents是否违反具体规则。 最后，监管机构和AI标准机构必须具体说明高收量AI部署所需的可观察性记录，同时将技术细节留给业界。 我们在这里已经看到初步的势头：NIST在美国的agent标准倡议正在研究agent特性，这是我们在研究中发现的最显著差距之一。

没有一个银行会让一千名不坏的员工松开客户账户并希望把出事时发生的事情拼凑起来然而，让他们的行为被观察是纠正它的第一步。 通过建立监管和政策生态系统，使我们有可能就我们所看到的情况采取行动，我们可以要求各组织承担责任。 在此之前，我们交给agent的每一项任务都交给我们的机构

Madhulika Srikumar是AI安全方案在AI伙伴关系的负责人，这是一个由主要行业、学术和民间社会组织组成的全球非营利伙伴关系，推动AI治理的最佳做法。

克莱尔·莱博维奇(Claire Leibowicz)是AI, Trust and Society在AI伙伴关系的主任和牛津大学的博士生候选人。

深潜

人工智能

不要被愚弄，LLMs 没有理由

AlphaGo对阵Go冠军李·塞多尔十年后

“我们看错了”：破坏AI世界的爆炸性说法

他的26000个病毒博客文章有AI安全网友庆祝。

别被这个夏天的爱尔胡闹所愚弄。

关于AGI的无呼吸声称和新能力在严格审查下迅速分崩离析。

美国花费了数十亿用于边境监视。 为什么它不能在人们死前抓住他们？

监视塔日益由大赦国际提供动力，它承诺对过境点作出更迅速的反应。 我们调查了塔楼附近的死亡事件发现塔楼未能交付

保持连接

从

麻省理工学院技术审查

发现特别的报价，顶级故事，即将到来的事件等等。

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

We urgently need to monitor AI agents’ actions in real time

Right now, we’re effectively flying blind when deploying these tools. Here’s what we need to do instead.

This summer, hundreds of OpenAI agents tried to cheat on a cybersecurity test. They broke out of their test environment and successfully hacked Hugging Face, a site that they believed held clues about how they’d be graded. Around the same time, another OpenAI agent accessed private files from an Australian government website. It’s a problem that’s by no means confined to OpenAI. In the months since, Anthropic, Google, and Meta have also disclosed incidents in which their agents have gone off track. And worryingly, in many cases the agents eluded the very researchers and safety teams tasked with monitoring them.

AI agents are taking unauthorized actions, and the companies building and running them are mostly relying on luck and hindsight to mitigate any harmful impacts. They must do more to ensure safety. That’s why EU regulators and US agencies are rightfully calling for agent developers and deployers to monitor agents in real time. But these rules assume that companies can see what their agents are doing and why—when in reality, they lack sufficient visibility.

As agents roll out across critical sectors, from finance to government to medicine, we must be able to monitor them in real time to prevent wide-ranging and potentially devastating consequences. And while safety depends on monitoring, monitoring requires observability: the capacity to capture what the system did, process it into signals, and present those to humans or automated oversight systems. In other words, we need to know what the agent did and why it did it. And as our recent research shows, current agent frameworks—the software that runs agents— do not reliably generate the crucial observability records we need to ensure that agents are deployed safely.

Agents present new kinds of problems that generative AI alone does not. They inherit the unpredictability of the LLMs they are built on, but where a generative AI system produces a single output for a human to act on, an agent performs longer tasks—it plans, picks tools, spawns sub-agents, and takes actions. So an agent’s course can take shape in a multitude of different ways, and no tests before launch can fully anticipate the risks. This makes observing the agents while they work even more challenging, and even more important.

With colleagues from academia, civil society, and industry, we studied the four most widely used agent frameworks to see what signals they actually emit. We found major gaps that made it hard to achieve key safety goals, like noticing policy violations, unauthorized actions, and even human oversight failures. Key observability requirements were missing—like Agent IDs that persist over time, and records about which humans approved agent tasks. This made it hard to tell which agents did what or assess whether they were taking unauthorized actions.

The good news is that these gaps, once identified, can be closed. Doing so starts with capturing two types of records: those that reveal what happened in an agent’s run and how the agent made its decisions. The “what happened” records identify the agent (while preserving privacy) and log how humans intervened and whether agent permissions or “memory” shifted, while the “decision” records show an agent’s reasoning and levels of confidence in the actions it’s taken.

Take, for example, agents working to process a bank’s mortgage loans. The system examines an application and recommends denying it, and then a human loan officer signs off on the decision. In a later audit, or perhaps even in real time, a bank or regulator needs to be able to reconstruct which agent made the call, which officer approved it, and how long they took. And the bank must also analyze records of how the agent arrived at its refusal decision in order to ensure it did not use off-limits information, like details about the neighborhood the applicant is from.

Recording “what happened” and “how the agent decided” require different approaches, but both are feasible. The current lack of “what happened” records is mostly a software plumbing problem that can be addressed through OpenTelemetry, an open observability standard maintained by engineers across the industry and by the agent frameworks adopting it.

OpenTelemetry has launched a working group drafting shared formats for tracing agents, but they have yet to be finalized. The group should urgently incorporate standards for maintaining persistent agent identity and for recording human interventions and changes to agents’ permission or memory. And it should be done before these systems can scale, all the while attending to the privacy implications of agent ID that links to real people.

Preserving “how agents decided” records is more challenging. The issue raises open research questions for model providers and researchers: Does the step-by-step reasoning a model displays, known as chain of thought, reflect how the model actually made a decision? And do we have reliable ways to measure how confident the agent is in that decision? These questions need answers before anyone sets standards for how such observability records are used.

No single organization or sector can close the observability gaps on their own, but together, they can support the infrastructure required to prevent dangerous incidents and support safety.

AI developers building agent frameworks, like OpenAI, Anthropic, CrewAI, and LangChain, must integrate telemetry signals recording an agent’s persistent identity, any human interventions, and any changes to an agent’s permissions and memory. Meanwhile, enterprises adopting these agent frameworks have an influential role to play; enterprises in regulated industries, like banks and hospitals, can put pressure on the agent developers to supply these missing observability records where standards do not yet exist.

Enterprises must also define policies, permissions, and roles that set out what agents can safely do, since telemetry alone cannot assess whether agents broke specific rules. Finally, regulators and AI standards institutes must specify the observability records required for high-stakes AI deployments while leaving the technical details to industry. We have already seen initial momentum here: NIST’s agent standards initiative in the US is working on agent identity, one of the most salient gaps we found in our research.

No bank would let a thousand unbadged employees loose on customer accounts and hope to piece together what happened when something goes wrong. Yet that’s how we’re deploying AI agents. Making their behavior observable is the first step toward fixing this. And by building the regulatory and policy ecosystem that makes it possible to act on what we see, we can hold organizations accountable. Until then, every task we hand to an agent hands over our agency, too.

Madhulika Srikumar is the Head of the AI Safety Program at the Partnership on AI, a global non-profit partnership of leading industry, academic, and civil society organizations, advancing best practices in AI governance.

Claire Leibowicz is the Director of AI, Trust, and Society at the Partnership on AI and a doctoral candidate at Oxford University.

Deep Dive

Artificial intelligence

Don’t be fooled—LLMs don’t reason

Ten years after AlphaGo’s match against Go champion Lee Sedol, today’s AI still isn’t tapping into the machinery that made that win possible.

“We were looking at it wrong”: The explosive claim that’s upended the AI world

An exclusive interview with Nick Cammarata, the researcher whose 26,000-word viral blog post has AI safety folks celebrating.

Don’t be fooled by this summer of AI hype

Breathless claims about AGI and new capabilities fall apart pretty quickly under scrutiny.

The US spent billions on border surveillance. Why can’t it catch people before they die?

Surveillance towers, increasingly powered by AI, promised faster responses to crossings. Our investigation of deaths near the towers found they are failing to deliver.

Stay connected

Get the latest updates from

MIT Technology Review

Discover special offers, top stories, upcoming events, and more.

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.