思想领袖
AI Agents 罕见崩溃。 他们静静的失败了研究显示为什么
在 Google 上的首选来源中添加 Unite.AI
大型语言模型agents现在回答支持票，查询数据库和计划出行。 当它们出错时，它们通常会返回一个自信的答案和成功的代码。 越来越多的研究解释了为什么这些失败很难看到，以及工程团队是如何学习测量这些失败的。
似乎成功的失败
传统软件往往会大声地失败。 一个函数丢出一个例外，一个服务器返回一个错误代码，一个监测仪表板会变成红色。 AI agent行为不同。 这是一个循环：一个语言模型读取了快取，决定调用搜索API或数据库等工具，读取结果，并决定下一步做什么。 每一步都可能出错，而链条中的每个网络请求仍然成功。
考虑一个普通案件。 天气工具失败，在正常反应中返回“ 无法提供的服务 ”。 该模型阅读了该文本，礼貌地道歉，或者更糟的是，它从培训数据中写出一个可信的预测。 用户收到流利的回答。 服务器日志以正常的空闲度显示成功请求。 常规监控库里没有任何记录显示出事了
模型提供者在自己的文件中描述了同样的空白。 兹克 1Z的Gemini API指导指出，该模型"不执行函数本身". 它只提出函数名称和参数，开发者的应用程序运行调用。 因此，该应用程序是系统唯一看到实际结果的部分。 Anthropic的"建造agents工程指南"提出了相关的观点，即agents在每一个步骤上都需要来自环境的"地上真理"，如工具结果，来判断自己的进度。 当一个工具将失败报告为普通文本时，这种地底真理是错误的，而agent是无法注意到的。
这不是一个新问题，只是旧问题的更尖锐的版本。 2015年由D. Sculley所领导的Google工程师在机器学习系统中描述了"隐藏的技术债务". 他们认为，生产ML系统中的大多数风险都位于模型代码之外，在数据依赖性，配置和胶体代码中。 Agents在该列表中新增一层。 提示和工具描述是可以改变系统行为而不改变其代码的配置。
小编辑，大效果
一个小的即时改变能有多重要？在ICLR 2024的论文中，华盛顿大学的梅兰妮·斯克拉尔和同僚只改变了几发提示的格式：分隔符，间隔符和大写符。 意义不变。 同一项任务的性能在LLaMA-2-13B上仍然有76个精确点的差异。 灵敏度并没有随着更大的模型，更多的实例或指令调谐而消失。
作者们建议报告各种可能迅速的格式，而不是一个数字。 对于工程团队来说，实用的教训比较简单。 即时编辑即为代码更改。 它需要一个版本历史，需要测试才能到达用户。
这同样适用于工具描述，模型读作是其快取的一部分。 Anthropic的指南建议，工具定义应该像主要的“迅速”工程注意。 它描述了一个用相对文件路径不断出错的编码agent. 当工具被更改为需要绝对路径时，模型使用时没有出错。 模型没有改变。 只有它读到的描述
一个成功证明小
第二行研究审视一致性。 基准测试由姚顺尧和同僚在Sierra发布，并提交ICLR 2025，模拟客户服务对话，其中agent必须使用工具并遵循业务规则。 作者推出一个称为pass^k的度量：agent在所有 k 尝试中正确解决相同任务的概率。
结果是清醒的。 GPT-4o agent在一次尝试中解决了大约61%的零售任务。 当它不得不连续八次成功完成同样的任务时，这个比例下降到了25%左右。 一次称的演示是关于一个系统在千回跑中的表现
多Agent系统中断之处
许多小组现在把几个agents连在一起，一个规划，另一个执行，第三个检查工作。 UC Berkeley的一项2025年研究“为什么多Agent LLM系统失败？” 分析了7个受欢迎多agent框架的1 600多条附加说明的行刑痕迹。 作者由梅尔特·克姆里(Mert Cemri)为首，构建了名为MAST的分类学，有14个故障模式，分为三个组：系统设计问题，agents之间的错配，以及任务验证。
他们的核心发现是，许多失败来自于系统的设计方式，而不仅仅是基础模型的极限。 Agents忽略了他们指定的角色，失去了对话的轨迹，或者宣布一项任务没有检查结果就完成了。 这些失败没有留下任何错误。 只有有人能读懂每个agent实际所做的，它们才可见。 兹克 2Z对多agent悖论的覆盖使得架构产生了相關的一分：增加agents可以放大一个早期出错而不是取消。
让看不见的可见
工程反应取自分布式系统，许多服务长期以来都遵循这些系统的要求，采用一种称为追踪的技术。 应用到agents，每个用户请求都成为了追踪，每个模型调用或工具调用都成为其中"span". 一个跨度记录了它的输入，输出，使用的符号以及花费了多长时间。 这项工作的行业标准机构OpenTelemoter正在起草关于基因AI跨度的共享公约。 截至2026年，这些公约仍然在发展过程中标出，因此属性名称仍然可能改变。
一些开源项目如Langfuse,Arize Phoenix和AcruxCore已经应用了这个想法。 他们的源代码是公开的，因此团队可以检查一个追踪是如何存储的并运行在自己的服务器上。 他们将模型呼叫和工具呼叫记录为单一跟踪的跨度，让团队给他们附加质量分数。 工具的选择不只涉及一个细节：一个工具跨度必须记录工具实际返回的内容，而不仅仅是网络请求是否成功。 没有这些，上述的气象工具故障就看不见了。
工业调查表明，现在追查已很普遍，但检测落后。 LangChain的Agent工程状况报告在2025年末对1,340名从业人员进行了调查。 质量是将agents投入生产的最大障碍，由32%的受访者命名。 89%的组织报告了对agents的某种可观察性，62%的组织详细追踪了个别步骤和工具呼叫。 只有52.4%在测试机上进行了下线评价，37.3%对现场流量进行了评价。 大多数球队可以看到他们的agents做了什么。 少检查是否正确。
分级回答
追踪显示发生了什么他们没有说答案是否很好。 手取阅每一次谈话都不进行分级，因此球队越来越多地使用第二个模型作为分级员，这种方法被称作"LLM-as-a-judge". 由梁明正和同僚所著的"NeurIPS 2023"一文发现，GPT-4等强力模型在超过80%的时间里都同意了人类的偏好，大致与人类彼此同意的速度相同。
同一份文件记录了该方法的弱点。 裁判模型更喜欢先显示的答案，更喜欢更长的答案，并且可以偏好自己所写的文字。 因此，法官的评分是估计数。 它们对于体积有用，但需要对照一小组被人类标注的例子加以检查，并随机排列答案顺序。
从实践到政策
这一衡量工作也正在成为治理方面的预期。 美国国家标准与技术研究所于2023年1月公布了AI风险管理框架。 它的四项核心职能之一，即“计量”，要求各组织随着时间的推移量化、跟踪和记录AI风险。 对agent来说，追踪和评分是函数要求的证据。
团队现在可以做什么
这与在这个平台上提出的可靠性是对代理AI的真正测试的论点相呼应。 新增上面的研究这个论点下面的测量细节：可靠性并不是你曾经观察到的属性，它是你必须持续测量的范围。
研究指出了一套很短的习惯，主要的模型提供者推荐了许多相同的习惯。 OpenAI建造agents的实用指南告诉团队在优化前要"设置evals以建立性能基线"，当agent超过一系列失败尝试或面临高风险行动时，要将控制权交给人类。 Anthropic建议在agents自行行动之前“在沙箱环境中进行广泛试验”。
实际上，这意味着几个具体步骤。 将提示和工具描述置于版本控制之下，作为代码。 对照固定数据集测试每个即时变化，并期望得出一系列结果而不是一个数字。 运行每个测试箱多次，因为一个通过会隐藏不一致。 记录工具输出在痕迹内，所以一个失败的工具不能隐藏在成功代码后。 将自动法官的分数作为估计，并对照人来检查。
这些习惯都不需要模型能力的突破。 它们要求将AI agent当作它是什么：一种必须测量的概率系统，因为它很少会宣布自己的失败。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Thought Leaders
AI Agents Rarely Crash. They Fail Quietly, and Research Shows Why
Add Unite.AI to your preferred sources on Google
Large language model agents now answer support tickets, query databases and plan trips. When they go wrong, they usually return a confident answer and a success code. A growing body of research explains why these failures are hard to see, and how engineering teams are learning to measure them.
A Failure That Looks Like Success
Traditional software tends to fail loudly. A function throws an exception, a server returns an error code, and a monitoring dashboard turns red. An AI agent behaves differently. It is a loop: a language model reads a prompt, decides to call a tool such as a search API or a database, reads the result, and decides what to do next. Each step can go wrong while every network request in the chain still succeeds.
Consider a common case. A weather tool fails and returns the text “service unavailable” inside an otherwise normal response. The model reads that text, apologises politely, or worse, writes a plausible forecast from its training data. The user receives a fluent answer. The server logs show a successful request with normal latency. Nothing in a conventional monitoring stack records that anything went wrong.
Model providers describe the same gap in their own documentation. Google’s Gemini API guide notes that the model “doesn’t execute the function itself”. It only proposes a function name and arguments, and the developer’s application runs the call. The application is therefore the only part of the system that sees the real result. Anthropic’s engineering guide on building agents makes the related point that agents need “ground truth” from the environment at each step, such as tool results, to judge their own progress. When a tool reports a failure as ordinary text, that ground truth is wrong, and the agent has no way to notice.
This is not a new kind of problem, only a sharper version of an old one. In 2015, Google engineers led by D. Sculley described “hidden technical debt” in machine learning systems. They argued that most of the risk in a production ML system sits outside the model code, in data dependencies, configuration and glue code. Agents add a new layer to that list. Prompts and tool descriptions are configuration that can change a system’s behaviour without any change to its code.
Small Edits, Large Effects
How much can a small prompt change matter? In a paper presented at ICLR 2024, Melanie Sclar of the University of Washington and colleagues changed only the formatting of few-shot prompts: separators, spacing and capitalisation. The meaning stayed the same. Performance on the same task still varied by up to 76 accuracy points on LLaMA-2-13B. The sensitivity did not disappear with larger models, more examples or instruction tuning.
The authors recommend reporting a range of performance across plausible prompt formats rather than one number. For engineering teams, the practical lesson is simpler. A prompt edit is a code change. It needs a version history, and it needs a test before it reaches users.
The same applies to tool descriptions, which the model reads as part of its prompt. Anthropic’s guide advises that tool definitions deserve “just as much prompt engineering attention” as the main prompt. It describes a coding agent that kept making mistakes with relative file paths. When the tool was changed to require absolute paths, the model used it without errors. The model did not change. Only the description it read did.
One Success Proves Little
A second line of research looks at consistency. The τ-bench benchmark, published by Shunyu Yao and colleagues at Sierra and presented at ICLR 2025, simulates customer-service conversations in which an agent must use tools and follow business rules. The authors introduced a metric called pass^k: the probability that an agent solves the same task correctly in all k attempts.
The results were sobering. A GPT-4o agent solved about 61% of retail tasks on a single attempt. When it had to succeed on the same task eight times in a row, the rate fell to about 25%. A demo that works once says little about how a system behaves on the thousandth run.
Where Multi-Agent Systems Break
Many teams now chain several agents together, with one planning, another executing and a third checking the work. A 2025 study from UC Berkeley, “Why Do Multi-Agent LLM Systems Fail?”, analysed more than 1,600 annotated execution traces across seven popular multi-agent frameworks. The authors, led by Mert Cemri, built a taxonomy called MAST with 14 failure modes in three groups: system design issues, misalignment between agents, and task verification.
Their central finding is that many failures come from how the system is designed, not only from the limits of the underlying model. Agents ignore their assigned role, lose track of the conversation, or declare a task finished without checking the result. These failures leave no error behind. They are visible only if someone can read what each agent actually did. Unite.AI’s coverage of the multi-agent paradox makes a related point from architecture: adding agents can amplify an early error rather than cancel it out.
Making the Invisible Visible
The engineering response borrows from distributed systems, where requests have long been followed across many services with a technique called tracing. Applied to agents, each user request becomes a trace, and each model call or tool call becomes a “span” inside it. A span records its input, its output, the tokens used and how long it took. The industry standards body for this work, OpenTelemetry, is drafting shared conventions for generative AI spans. As of 2026, those conventions are still marked as in development, so attribute names may still change.
Several open-source projects such as Langfuse, Arize Phoenix and AcruxCore already apply this idea. Their source code is public, so teams can inspect how a trace is stored and run the software on their own servers. They record model calls and tool calls as spans of a single trace and let teams attach quality scores to them. The choice of tool matters less than one detail: a tool span must record what the tool actually returned, not only whether the network request succeeded. Without that, the weather-tool failure described above stays invisible.
Industry surveys suggest that tracing is now common, but testing lags behind it. LangChain’s State of Agent Engineering report surveyed 1,340 practitioners in late 2025. Quality was the most cited barrier to putting agents into production, named by 32% of respondents. 89% of organisations reported some form of observability for their agents, and 62% had detailed tracing of individual steps and tool calls. Only 52.4% ran offline evaluations on test sets, and 37.3% evaluated live traffic. Most teams can see what their agents did. Fewer check whether it was correct.
Grading Answers at Scale
Traces show what happened. They do not say whether the answer was good. Reading every conversation by hand does not scale, so teams increasingly use a second model as a grader, an approach known as LLM-as-a-judge. A NeurIPS 2023 paper by Lianmin Zheng and colleagues found that strong models such as GPT-4 agreed with human preferences more than 80% of the time, about the same rate at which humans agree with each other.
The same paper documented the method’s weaknesses. Judge models prefer the answer shown first, prefer longer answers, and can favour text written by themselves. Judge scores are therefore estimates. They are useful for volume, but they need to be checked against a small set of human-labelled examples, with answer order randomised.
From Practice to Policy
This measurement work is also becoming a governance expectation. The US National Institute of Standards and Technology published its AI Risk Management Framework in January 2023. One of its four core functions, “Measure”, asks organisations to quantify, track and document AI risks over time. For an agent, traces and evaluation scores are the evidence that function asks for.
What Teams Can Do Now
This echoes the argument made on this platform that reliability is the real test of agentic AI. The research above adds the measurement detail underneath that argument: reliability is not a property you observe once, it is a range you have to keep measuring.
The research points to a short set of habits, and the major model providers recommend many of the same ones. OpenAI’s practical guide to building agents tells teams to “set up evals to establish a performance baseline” before optimising, and to hand control to a human when an agent exceeds a set number of failed attempts or faces a high-risk action. Anthropic recommends “extensive testing in sandboxed environments” before agents act on their own.
In practice, that means a few concrete steps. Keep prompts and tool descriptions under version control, as code. Test each prompt change against a fixed dataset, and expect a range of results rather than one number. Run each test case several times, because one pass hides inconsistency. Record tool outputs inside traces, so a failed tool cannot hide behind a success code. Treat automated judge scores as estimates, and check them against people.
None of these habits require a breakthrough in model capability. They require treating an AI agent as what it is: a probabilistic system that must be measured, because it will rarely announce its own failures.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.