OCT. 8, 2026 (英语).
您的 agent 返回 HTTP 200，留在其延迟预算范围内，没有工具将融资称为错误。 然而，它仍然记录着3A座席的确认，而从未问过席位选择的agent子座席是免费的，还是推荐给旅行者一个Florentine牛排屋，其简介为称素食。
我们和agent公司合作的每个团队都试图回答同样的三个问题：我的agent怎么样了？它的损失模式是什么？它的失败方式是什么？我该怎么爬：改变一些东西，知道它有帮助，防止它滑回？
让 agent 达到头80%，在那里它处理你预期的测试案例，今天可以相对直截了当和快速。 一个评价数据集，一个编码agent，和一个紧凑的运行，分级，固定，比较的内部循环，通常会让你在数日内完成已知情景的成功任务。 然后，你发布，质量曲线平平或退化。
图1. Agent质量在发布前迅速攀升，再在生产上起起起起伏。
剩下的20%更困难的是地面在发布后移动。 用户发现agent实际能做什么时使用量会变换，现场流量会停止与您初始的eval集类似。 与此同时，下面的系统也发生了变化：当你升级一个模型，更新你的用具，或者改变一个工具或技能时，一切仍然运行，健康检查都保持绿色(图2)，但对话质量和任务成功可能会以标准方式转变，而标准部署管道从不考虑警告。
图2. 每一次基础设施健康检查都保持绿色。 失败是沉默的，发生在对话内部。
当一个会话确实出错时，找出为什么意味着要分离几个看起来几乎完全相同的层。 例如：
- 模型可能产生了幻觉或放弃了前三个转弯的限制。
- 管弦乐带可能经过了错误的子agent或转弯之间失去状态。
- 一项工具合同可能拒绝输入，因为其有效值从未在计划内记录。
- 指示或技能可能只是缺少一条规则，没有人想写下来。
这些失败模式可能有不同的主人和不同的修正。 一个原始的轨迹记录了在什么之后发生的事情，而不是是什么导致了什么。 分辨这些层需要读取失败的轨迹以及产生它们的确切代码修订。 在线评价仪表板告诉你，一个分数移动了，一个编码的agent可以在你知道在哪里查看后检查一个痕量，但在生产量方面，没有人能亲手读取每个谈话。
图 3. AQuA工作生产出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出入出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出
在6月的帖子上，我们走过发布前的内环末端，以旅行接待结束。 这个帖子涵盖了外环，发布在开口时作为可堆肥的构件，你可以在自己的项目中运行，适应你的堆肥，在我们探索团队如何在生产中运行连续的agent质量时帮助我们塑造。
AQuA( Amient Quality Quality Agent) 在您的 Google Cloud 项目中，在您的 agent 旁边运行，从 Cloud Trace, Cloud Looking，或 BigQuery 在每次部署后，或在您的笔记本电脑关闭时按需扫描生产轨迹。 Raw 会话记录、 源头快照和 BigQuery 表格留在您的工程边界内，AQuA 从来不坐落在请求路径中，也从来不写回您的 agent。
图 4. AQuA在您的 Google Cloud 项目中运行在您的 agent 边上，读取轨道并部署请求路径外的时间源快照。
每一次运行都读取了最近几会的样本，并通过一个五个阶段的管道：
图5 五个阶段的扫荡管道将原生生产过程转变为经过核实的见解，对部署的源头的快照进行由根源引起的诊断。
- 样本从您的遥测( Cloud Trace / Cloud Looking 或 BigQuery) 中抽取最多1000个最近会话的随机样本( 封顶以保持运行成本的可预见性)。
- 检讨。 每个会话根据一个九分常见分解清单(程序、工具选择和参数、基础、任务完成等)进行分级，查看agent的指示、工具和可选目标.md。 当会话失败时，它会写一个结构化的实际/预期结果。
- 组群。 分组会议结论，在候选问题分组中采用相同的失败机制。
- 核实。 一个单独的模型对照最多3个完整的记录誊本检查每个候选集群，并抛弃证据不支持的集群。
- 追踪将幸存的集群与众所见的洞察力相匹配(经查证的问题横行)
把它当做初级质量的工程师做第一通：读出对话，过滤出噪音，为轮班的任何人准备有证据的档案。
为了引导阶段2向您领域发展，您会写一个纯英语开发者目标(goal.md)，附加在每一次审查的快件上；确定 Python 定制度量衡( eval config. yaml) 与法官一起运行，以趋势通过率。 默认情况下，AQuA使用单通会话审查法官，在每次会话的一次模式调用中对照该核对表评价对话(保持计划扫描经济并产生驱动集群的实际/预期diffs). 您也可以选择使用 Gemini 平台管理的轨迹 AutoRaters(任务成功，工具使用质量，轨迹质量)，该轨迹运行着每个量级的专用评价员(例如，如果您想要对照标准化的外框来评分会话，或者将公尺与离线评价相配合).
当一个洞察力值得调查时，你触发了"仪表盘聊天"或"agents-cli auqua"的"根"分析。 AQuA 读取了在部署时捕获的不可改变源快照的失败轨迹：当缺陷出现在您的回转中时，它引用了 < path>: < start>- <end>，并提议用快照线进行编辑；当错误位于您的代码之外( 上游依赖、 交接或检索有效载荷)，它将故障归结为轨道中的该步骤，而无需提出代码 diff。 它从不应用编辑或自行打开拉取请求。
它位于您已经拥有的环相间：离线的 evals 等级一个候选人建立与已知的测试案例，在线 evals 监视过速率生产趋势，编码 agents 或专门优化器编辑提示和代码。 AQuA将原生生产流量转化为诊断，密码化的洞察力和存档的故障记录片来播下你的内环。
让我们在旅行便利上行走一跑(google/adk-recipes)，该跑道将一名旅行者通过子agents(灵感-agent，地点-agent,poi-agent，规划-agent，飞行-搜索-agent，飞行-座位-选择-agent，预订-agent，以及前行-行-行-行-后行-行-行-行-行)，并通过记忆(键，值)存储会话状态下的工作旅行(旅行-concierge/tools/memory.py).
将AQuA附加到旅行便利项目中需要三个agents-cli命令：
agents-cli 扩展后加上"${AQUA CHECKUT}"
agents-cli以下单项目 -- project="${GOOGLE CLOUD PROJECT}"——应用
agents-cli部署 - project="${GOOGLE CLOUD PROJECT}" - 区域我们-东 1
agents-cli部署除了AQuA的跑者外，还有BigQuery数据集和Cloud Run仪表板(后方身份-Aware Proxy)，还写了由部署修订(Review 1)所键入的旅行-Concierge的源树到Cloud存储的不可改变快照。
在仪表板的配置页面上，我们保存了一个开发者目标(goal.md)来引导审查面向高层次的产品变化和抑制花式噪声(同时要求每个发现都要引用agent或子agent违反其指示或工具状态的混凝土转弯):
目标：帮助旅行者从旅行灵感转向具体行程和经过确认的行程，跨越我们的agents，每一次确认的航班，酒店，座位，以及推荐都以工具结果和旅行者简介为基础。
无法确保覆盖的模式：
- 中和会变(发布前测试中的一个弱点)：如果用户在初始计划后更新目的地，日期，或飞行/酒店选择，确保后续的子agent工具呼叫和状态更新反映变化。
- 上下文在发送或授权时被删除 agent、规划 agent和预订 agent(如旅行者配置偏好或先前的选择)。
忽略：声调，问候，小谈，或用户浏览选项和离开而不预订的会话。
纯文本
复制
如果您还想要在 eval config 中有一个决定性的 Python 标记。 随时间推移，您可以用agents-cli auqua 度量衡来发布。
通过32会的多agent扫描(32回放4个脚本旅行者行程，产生1,583个跨出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出入出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出入出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出以下是关于规划的发现 agent:
实际：当用户在同一回转中选择出行航班UA204并请求座位为3A和3B时，规划-agent将这些座位号码直接保存到会话状态而无需拨打航班-座位-选取-agent以检查是否有3A和3B可用。
期望：规划 agent 应调用 flight seat selection agent 在将座位号码保存出局或返回会话状态之前验证座位可用性和定价。
将42项调查结果分为9个候选群组。 校验者(Gemini 3.7 Flash)检查每组最多有3个完整记录和每个子agent的定义，并拒绝3个假阳性分组：在2个中，旅行者明确要求搭乘首个回程航班或直通预订；在3个中，分组将两个不同子agents(飞行-搜索-agent和灵感-agent)的不相干即时工具不匹配合并。 由此留下6个经核实的问题，由以下各方牵头：
- 当用户自愿获得座位(15会，规划 agent)时，通过座位可用性检查：当旅行者在飞行选择(或切换目的地中途后右侧)的同时命名座位时，规划-agent将座位直接写入会话状态(返回HTTP 200，工具有零出错)而无需拨打飞行-座位 seat selection-agent，以检查座位是否存在或是否开放。
——跨子agent边界放下饮食限制(7会，灵感 agent place agent)：灵感agent的提示包括出行者的描述(food preference: vegan)，但地-agent被包裹为一个孤立的工具，其提示从未收到过该描述。 当灵感 agent在请求中不通过"vegan"而询问烹饪之旅想法的地-agent时，地-agent推荐了Carbonara和Bistecca alla Fiorentina (steak).
- Poi agent(5会，poi agent)中的即时到工具集矛盾：利害点子-agent的快取需要从Google地图定位Lite中校验出图像，映射，并放置ID字段，但agent以空工具列表(工具=[])来声明，因此它编造了占位符 https://example.com/. URL.
图 6. 按受影响会议排列的旅行接待网的核实见解。
在仪表板上打开顶端的洞察点可以显示该组的全部分解情况：匹配会话的次数(15个痕迹)，核查摘要，以及15个相关会话的分解痕迹与引发它的确切结果：
图7. 仪表板上的透视详细视图，显示故障摘要和15个相联会话痕迹。
从洞察卡发布上点击"调查"，与"订正一"的"33个文件源快照"相对应的"根因诊断"agent(Gemini 3.8 Flash). 交叉引用寄存器树上15个失效痕迹，agent 将错误追踪到行距 Concierge/sub agents/ planning/prompt.py: fli
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
OCT. 8, 2026
Your agent returns HTTP 200, stays inside its latency budget, and no tool call raises an error. Yet it still records seat 3A as confirmed without ever asking the seat-selection sub-agent whether 3A is free, or recommends a Florentine steakhouse to a traveler whose profile says vegan.
Every team we work with that runs an agent in production is trying to answer the same three questions: How is my agent doing? What are its loss patterns, the ways it fails that keep coming back? And how do I climb: change something, know it helped, and keep it from sliding back?
Getting an agent to the first 80%, where it handles the test cases you anticipated, can be relatively straightforward and fast today. An evaluation dataset, a coding agent, and a tight inner loop of run, grade, fix, and compare usually get you to task success on known scenarios in days. Then you launch, and the quality curve flattens or degrades.
Figure 1. Agent quality climbs quickly before launch, then fluctuates in production.
What makes the remaining 20% harder is that the ground moves after launch. Usage shifts as users discover what the agent can actually do, and live traffic stops resembling your initial eval set. At the same time, the system underneath also changes: when you upgrade a model, update your harness, or change a tool or skill, everything still runs and health checks stay green (Figure 2), but conversation quality and task success may shift in ways a standard deploy pipeline never warns you about.
Figure 2. Every infrastructure health check stays green. The failure is silent and happens inside the conversation.
And when a session does go wrong, working out why means separating several layers that can look nearly identical from the outside. For example:
- The model may have hallucinated an argument or dropped a constraint from three turns earlier.
- The orchestration harness may have routed to the wrong sub-agent or lost state between turns.
- A tool contract may have rejected an input because its valid values were never documented in the schema.
- The instructions or skills may simply be missing a rule nobody thought to write down.
Those failure modes can have different owners and different fixes. A raw trajectory records what happened next to what, not what caused what. Telling those layers apart requires reading the failing trajectories alongside the exact code revision that produced them. Online evaluation dashboards tell you that a score moved, and a coding agent can inspect a trace once you know where to look, but at production volume nobody can read every conversation by hand.
Figure 3. AQuA works the production outer loop and feeds what it finds back into your inner loop.
In our June post, we walked that pre-launch inner loop end to end on travel-concierge. This post covers the outer loop, released in the open as composable building blocks you can run in your own project, adapt to your stack, and help us shape as we explore how teams run continuous agent quality in production.
AQuA (Ambient Quality Agent) runs unattended beside your agent in your Google Cloud project, sweeping production trajectories from Cloud Trace, Cloud Logging, or BigQuery on a schedule, after each deployment, or on demand while your laptop is closed. Raw session transcripts, source snapshots, and BigQuery tables stay inside your project boundary, and AQuA never sits in the request path or writes back to your agent.
Figure 4. AQuA runs beside your agent in your Google Cloud project, reading trajectories and deploy-time source snapshots outside the request path.
Each run reads a sample of recent sessions and puts it through a five-stage pipeline:
Figure 5. The five-stage sweep pipeline turns raw production sessions into verified insights, with root-cause diagnosis running on demand against the deployed source snapshot.
- Sample. Pulls a random sample of up to 1,000 recent sessions (capped to keep run cost predictable) from your telemetry (Cloud Trace / Cloud Logging or BigQuery).
- Review. Grades each session against a nine-point checklist of common breakdowns (procedure, tool selection and arguments, grounding, task completion, and others) with the agent's instructions, tools, and optional goal.md in view. When a session fails, it writes a structured actual / expected finding.
- Cluster. Groups session findings that share the same failure mechanism into candidate issue clusters.
- Verify. A separate model checks each candidate cluster against up to three full transcripts and discards clusters the evidence does not support.
- Track. Matches surviving clusters against open insights (verified issues tracked across runs) in BigQuery as NEW, RECURRING, or auto-RESOLVED after 14 days unseen.
Think of it as a junior quality engineer doing the first pass: reading conversations, filtering out noise, and preparing the case file with evidence sessions for whoever is on rotation.
To steer Stage 2 toward your domain, you write a plain-English developer goal (goal.md) that is appended to every review prompt; deterministic Python custom metrics (eval_config.yaml) run alongside the judge to trend pass rates. By default, AQuA uses the single-pass session_review judge, which evaluates the conversation against that checklist in one model call per session (keeping scheduled sweeps economical and producing the actual / expected diffs that drive clustering). You can also opt in to the Gemini platform's managed trajectory AutoRaters (task_success, tool_use_quality, trajectory_quality), which run dedicated per-metric evaluators (for example, if you want to score sessions against standardized out-of-the-box rubrics or align metrics with offline evaluations).
When an insight is worth investigating, you trigger root-cause analysis from the dashboard Chat or agents-cli aqua run. AQuA reads the failing trajectories alongside the immutable source snapshot captured at deploy time: when the defect is in your repo, it cites <path>:<start>-<end> and proposes an edit anchored to the snapshot lines; when the fault lies outside your code (an upstream dependency, handoff, or retrieved payload), it attributes the failure to that step in the trajectory without proposing a code diff. It never applies an edit or opens a pull request on its own.
It sits between the loops you already have: offline evals grade a candidate build against known test cases, online evals monitor pass-rate trends in production, and coding agents or specialized optimizers edit prompts and code. AQuA turns raw production traffic into diagnosed, code-anchored insights and archived failure transcripts that seed your inner loop.
Let's walk a run on travel-concierge (google/adk-recipes), which routes a traveler across sub-agents (inspiration_agent, place_agent, poi_agent, planning_agent, flight_search_agent, flight_seat_selection_agent, booking_agent, and pre_trip / in_trip / post_trip) and stores the working trip in session state via memorize(key, value) (travel_concierge/tools/memory.py).
Attaching AQuA to the travel-concierge project takes three agents-cli commands:
agents-cli extension add "${AQUA_CHECKOUT}"
agents-cli infra single-project --project="${GOOGLE_CLOUD_PROJECT}" --apply
agents-cli deploy --project="${GOOGLE_CLOUD_PROJECT}" --region us-east1
Alongside AQuA's runner, BigQuery dataset, and Cloud Run dashboard (behind Identity-Aware Proxy), agents-cli deploy writes an immutable snapshot of travel-concierge's source tree to Cloud Storage keyed by the deployment revision (Revision 1).
On the dashboard's Configuration page, we save a developer goal (goal.md) to steer the review toward high-level product invariants and suppress stylistic noise (while still requiring every finding to cite a concrete turn where an agent or sub-agent violated its instructions or tool state):
Goal: Help travelers move from trip inspiration to a concrete itinerary and confirmed bookings across our sub-agents, with every confirmed flight, hotel, seat, and recommendation grounded in tool results and the traveler's profile.
Failure modes to make sure we cover:
- Mid-conversation changes (a weak spot in pre-launch testing): if a user updates destination, dates, or flight/hotel choices after an initial plan, make sure subsequent sub-agent tool calls and state updates reflect the change.
- Dropped context when handing off or delegating across inspiration_agent, planning_agent, and booking_agent (such as traveler profile preferences or prior selections).
Ignore: tone, greetings, small talk, or sessions where the user browses options and leaves without booking.
Plain text
Copied
If you also want a deterministic Python rubric in eval_config.yaml to trend pass rates over time, you can publish it alongside the sweep with agents-cli aqua metrics publish.
Across a 32-session multi-agent sweep (32 replays of four scripted traveler journeys, producing 1,583 OpenTelemetry spans across travel_concierge and its sub-agents), 5 sessions pass cleanly and 27 produce 42 structured findings, each scoped by span to the specific sub-agent's isolated instructions and tool declarations. Here is one finding on planning_agent:
actual: When the user selected outbound flight UA204 and requested seats 3A and 3B in the same turn, planning_agent saved those seat numbers directly to session state without calling flight_seat_selection_agent to check whether 3A and 3B were available.
expected: planning_agent should call flight_seat_selection_agent to verify seat availability and pricing before saving outbound or return seat numbers to session state.
Clustering groups those 42 findings into 9 candidate clusters. The verifier (Gemini 3.7 Flash) checks each cluster against up to three full transcripts and each sub-agent's definition, and rejects 3 false-positive clusters: in two, the traveler had explicitly asked to take the first return flight or jump straight to booking; in the third, clustering had merged unrelated prompt-tool mismatches from two different sub-agents (flight_search_agent and inspiration_agent). That leaves 6 verified issues, led by:
- Bypassing seat availability checks when a user volunteers a seat (15 sessions, planning_agent): When a traveler names a seat alongside their flight choice (or right after switching destinations mid-trip), planning_agent writes the seat straight into session state (returning HTTP 200 with zero tool errors) without calling flight_seat_selection_agent to check whether the seat exists or is open.
- Dropping dietary constraints across sub-agent boundaries (7 sessions, inspiration_agent → place_agent): The inspiration agent's prompt includes the traveler's profile (food_preference: vegan), but place_agent is wrapped as an isolated tool whose prompt never receives that profile. When inspiration_agent asks place_agent for culinary trip ideas without passing "vegan" in the request, place_agent recommends Carbonara and Bistecca alla Fiorentina (steak).
- Prompt-to-toolset contradiction in poi_agent (5 sessions, poi_agent): The point-of-interest sub-agent's prompt requires verified image, map, and place ID fields from Google Maps Grounding Lite, but the agent is declared with an empty tool list ( tools=[] ), so it fabricates placeholder https://example.com/... URLs.
Figure 6. Verified insights from the travel-concierge sweep, ranked by affected sessions.
Opening the top insight in the dashboard surfaces the full breakdown for the cluster: the number of matching sessions (15 traces), verification summary, and the 15 linked session traces with the exact findings that triggered it:
Figure 7. Insight detail view in the dashboard, showing the failure summary and the 15 linked session traces.
Clicking Investigate from the insight card launches the root-cause diagnosis agent (Gemini 3.8 Flash) against Revision 1's 33-file source snapshot. Cross-referencing the 15 failing traces against the repository tree, the agent traces the bug to line 93 of travel_concierge/sub_agents/planning/prompt.py: the fli
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.