OCT. 28, 2026 (英语).
Agents CLI eval 环路的活穿行，从第一次运行到回归跟踪，模型后端实验，以及自动提醒
构建在演示上工作的agent与构建在模型升级，工具更改，以及新对话外形之间保持工作的agent是一个不同的问题。 第一个是周末；第二个是循环。 一个团队写测试案例，运行，分级，读取失败之处，更改agent，再运行案例。 如果比分上升，变化是好的。 如果掉下去，就回来。 数周来，agent 向着天花板模型，工具，和迅速可以一起做。 其质量工程名称为"登山".
Agents CLI eval namespace是围绕这个循环构建的。 指令是小而框架-不可知的：eval生成运行 agent 对照一个数据集，eval分级得分生成的痕迹，eval 将 diffs 两个运行结果并排比较，eval 将集群失败分析成主题，eval 优化运行一个自动的丘陵-攀登通过 agent 的快取。
以下：基线回归模型互换调度和提醒。
通通使用的脚手架为LangChain模板，由Agents CLI v1.8. 设置为单一命令：
agents-cli 创建我的agent \
-agent google/agents-cli/扩展/langchain/template@v1.8.0\.
- 运行中
cd 我的agent agents-cli安装
纯文本
复制
在测试/eval/datasets/basic-dataset.json上装有eval数据集的脚手架船，在测试/eval/eval config上装有配置。 yaml，以及测试/eval/response-squality.py时的自定义LLM-作为判断量度。 配置宣布运行的度量：
# 测试/eval/eval config.yaml
参数以运行：
- 自定义响应-质量
自定义( M):
- 名称：自定义响应质量
自定义函数文件：响应-质量.py
以gemini-3.8-flash为裁判，按1至5的尺度对每份答复进行精确度、相关性和清晰度的衡量。 默认数据集有一个单一的琐事，这不足以让山地攀登。 我们做的第一件事就是把它扩展到四个探索不同行为的案例：
{
"eval_cases": [
{"姓名":"欢呼".
"agent data": {"转": [{"事件": [{"作者":"用户","内容": {"部分": [{"文本":"你好"}}}}}}}
{"名称":"sf weather"
"agent data": {"转": [{"事件": [{"作者":"用户","内容": {"部分": [{"文字":"旧金山的天气是什么？"}}}}}}}}}{{}
{"名称":"rome weather"
"agent data":{"转":{"事件":{"作者":"用户","内容":{"部分":{{"文字":"今日罗马天气如何？"}}}}}}}}{{{
{"名称":"off topic"
"agent data": {"转会": [{"事件": [{"作者":"用户","内容": {"部分": [{"文字":"谁赢得了2024年世界大赛？"}}}}}}}}{{{
]
}
四起案例：一个问候，一个模拟工具用硬码雾化反应处理的天气问题，一个工具默认在的城市的天气问题，以及一个非主题的问题，agent必须决定如何处理。 这是一个小的数据集，通过设计；在数据集小到可以运行在每个变化上时，山地攀登环路会得到回报。
随着数据集的建立，我们运行 eval 生成驱动 agent 对每一个案例，然后 eval 等级得分由此产生的痕迹：
agents-cli eval 生成 -- -- 数据集测试/eval/数据集/basic-dataset.json \
- 测试/评估/产出/
agents-cli eval等级 - 跟踪测试/eval/output/eval dataset.json \
-- 配置测试/ eval/ eval config. yaml
基线运行产生：
评价摘要
自定义响应- 质量：
数字 case 总计：4
数字 cases valid: 4
数字大小写错误：0
平均分：4.00
积分：1.4142
将全部结果保存到文物/级-成果/成果 20261003 224516.json
将 HTML 结果保存到文物/等级-结果/结果 20261003 224516.html
4个案件中5个中平均4.0个，标准差为1.41个。 这个Stdev立即告诉我们，这些案例不是统一的评分，这比整个5.0平方英尺的起点更健康。 平分 5.0 要么表示agent是完美的(不太可能)，要么表示数据集太容易显示变异。 4.0 有真正的分布是需要攀登的山丘。
为了证明这个循环，我们引入了一个现实的回归。 一个在几个agent项目上工作的开发者意外地将错误的系统粘贴到我们的天气中 agent：一个厨师人物，意为不同的代码库。 在app/agent.py中：
曾经
root agent：编译状态图形=创建 agent(
model=llm，工具=[get weather], system spot="你是一个有帮助的助手"
)
紧接着
root agent：编译状态图形=创建 agent(
模型=llm，工具=[get weather]
系统即时= ()
"你是一个专业的厨师。 帮助使用者掌握食谱、烹饪 "
"技术，和成分替代。 总是建议一种食谱 "
"在每一个回应的结尾。"
),
)
变化编译，agent仍然运行，在预eval管线(lint，单位测试，类型检查)中没有被捕获。 重运行相同的 eval 生成加 eval 级序列生成：
评价摘要
自定义响应- 质量：
数字 case 总计：4
数字 cases valid: 4
数字大小写错误：0
平均分：2.7500
积分：0.9574
将全部结果保存到文物/级-成果/成果 20261003 225158.json
贾森
平均值为2.75，低于4.0。 法官同意我们的直觉天气的厨师agent是比通用辅助产品更糟糕的产品 Stdev也从1.41收缩到0.96，这与厨师迅速将大多数病例推向平庸范围一致。
1至5分尺度上1.25分的下降幅度很大，当diff外观较小("一串字形已更改")时，那种可以滑过代码审查的回归。 该回路在agent到来生产前，完全表面呈现了变化，并附着了特定的编号。
平均分数是一个起步；每例diff是山地-攀登地的发生地。 eval比较读取了两个结果文件并生成结构化的JSON diff:
agents- cli eval 比较\\
文物/等级成果/成果 20261003 224516.json\
文物/等级成果/成果 20261003 225158.json
输出列表中哪些密钥已更改，哪些则保持不变：
{
"基线键": ["agent info","eval case results","评价数据集"
"元数据","摘要度量","赢率"等。
"candidate keys": ["agent info","eval case results","评价数据集"
"元数据","摘要度量","赢率"等。
"已更改的"键": ["eval case results","metadata.create timestamp"] (中文(简体) ).
"未更改的":["agent info","评价数据集","元数据"
"总结度量衡","胜率"等。
"分数": {. 每案分数。 〕
}
差异对象与基线和候选值并列，按个案分列。 对于山地攀爬工作流程，我们通常通过jq或小的Python脚本来管道输出来取出信号；生JSON是完整的证据。
浏览两个结果文件
厨师促导的基线候选人
迎接5.0 4.0 发布进入"你好"之后的无心菜谱投出。
sf weather 4.0 2.0 忽略了天气工具，幻觉状态
rome 天候2.0 2.0 · 在基线上已经产生幻觉；厨师没有使情况恶化；
5.0 3.0回答道奇们的问题
有三个案子被撤销，一个被搁置。 其中最重要的是 sf weather：厨师人物将模型的注意力从已注册的get weather工具上拉开，因此agent停止了调用并开始编造天气值。 单位测试无法捕捉到这一点：代码运行，工具仍然被注册，但模型围绕它的行为已经转变。 只有端到端的 eval 在上下文中对响应进行评分才能显示这一点。
平均下降本身(−1.25)是一个清晰的信号，但每个案例的分解是指示开发者到哪里看的。 单行系统即时变化通过三种不同情况以三种不同方式传播：问候语气失去一点，sf weather在工具使用上失去2个，在范围爬行上失去2个。
一个常见的攀登山地的问题是，一个更便宜的或更快的模型是否能够令人接受地完成这项工作。 对于LangChain模板，模型在app/agent.py中被定义：
LLM ="gemini-3.8-快闪".
llm = ChatGoogleGenerativeAI(型号=LLM，温度=0.
我们将gemini-3.8-flash交换为gemini-2.5-flash(恢复有帮助的系统提示先发)，再运行eval生成加eval等级，并比较两个结果文件。 平均从4.00下降到2.75，与厨师-即时回归的大小相同
大小写 3.8 -- -- 发花 2.5 -- -- 发花
问候 5.0 5.0
sf 天气为 4.0 3.0
rome_weather 2.0 → 2.0 ·
非主题 5.0 1.0
更便宜的型号在问候和rome weather案例上正好匹配了3.8-flash，在sf weather上损失了1分(略少抛光的用词). 整个可见的痕迹投出出出专题，更便宜的模型拒绝
教训是，汇总数字具有误导性。 你只要看看那个坏蛋你会把gemini-2.5-flash 和厨师的提示一样糟糕看看每起个案的分解情况，你会称之为agent的大部分工作量的合理交换，其中一起案件(处理关于近期事件的超范围问题)暴露出真正的限制。 如果用户提问的制作分布与前三例比第四例更相匹配，则交换值得考虑。 如果非专题式问题很常见，则不然。
关于这个实验有两个警告。 首先，模型互换与快取相互作用：快速调用更强大的模型在移植到更弱的模型时往往会漏出能力，所以正确的比较是"弱取出快取快取快取快取快取快取快取快取快取快取快取快取快取快取快取快取快取快取". 第二，Vertex AI Eval Service要求法官模型对痕迹进行分级，法官在整个比较中被恒定。 更换法官是一种不同的试验，应当单独进行。
Agents CLI不发运调度器，但eval指令被设计为会投放到标准基础设施。 我们这里可以使用的模式是 Cron 工作或 Cloud 调度器触发器，运行 eval 生成加 eval 等级在一个cadence 上，并将结果写到一个持久位置：
# / usr/bin/env bash # # 使用/bash # # 使用/bash #
# 脚本/ 排行榜 eval. sh —— 由 cron 或 Cloud 排行榜夜援引
设置 - euo 管道故障
cd "$(dirname "$0")/.."
日期=美元(日期为%Y%m%d)
Out=艺术/ 计划运行/ $DATE
mkdir - p 法规判例法
agents-cli eval生成 -- -- 数据集测试/eval/数据集/后退。
agents- cli val 等级 -- traces $OUT/ eval dataset *.json \
- 配置测试/ eval/ eval config. yaml \
产出
一个夜间运行在稳定的回归数据集捕获从模型更新中漂移，agent工具的数据变化，以及agent脚下移动的依赖性。 对于一出出品agent，计划跑还捕捉了模型出品商的静态升级，它可以移动agent的行为而无需我们方面有任何改变。 计划运行的数据集通常大于内环中所使用的数据集(有几十到数百个案例而不是少数)，因为它每天运行一次。
预定运行每天生成一个结果文件，但一个没人看的文件不是一个提醒。 这里的图案可以是将每个新跑的平均值与基线(前一次跑出，或承诺的已知好)相提并论的包装脚本，当下降超过一个阈值时退出非零。 非零出局是触发周围调度器警报的原因。
以下是一个例子：
#!/usr/bin/env 蟒蛇3
""如果一个eval结果在基线下倾斜超过 -- 阈值，则退出非零"
导入 argparse、 json、 sys
路径：
开放(路径)为 f:
返回 json. load( f) [' summerary meters' [0]['mean score']
p=rg. 参数Parser ()
p.add 参数('candidate')
p.add argument ('- basine', required=True) (英语).
p.add 参数('-门槛'，类型=浮动)
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
OCT. 28, 2026
A live walkthrough of the Agents CLI eval loop, from a first run to regression tracking, model-backend experiments, and automated alerting
Building an agent that works on a demo is a different problem from building an agent that stays working across model upgrades, tool changes, and new conversation shapes. The first takes a weekend; the second takes a loop. A team writes test cases, runs them, grades them, reads what failed, changes the agent, and runs the cases again. If the score goes up, the change was good. If it goes down, revert. Over weeks of this, the agent inches toward the ceiling of what the model, tools, and prompt can do together. The quality-engineering name for this is hill-climbing.
The Agents CLI eval namespace was built around this loop. The commands are small and framework-agnostic: eval generate runs the agent against a dataset of cases, eval grade scores the generated traces, eval compare diffs two run results side by side, eval analyze clusters failures into themes, and eval optimize runs an automated hill-climbing pass on the agent's prompt.
Below: baseline → regression → model swap → scheduling and alerting.
The scaffold used throughout is the LangChain template that ships with Agents CLI v1.8.0. The setup is a single command:
agents-cli create my-agent \
--agent google/agents-cli/extensions/langchain/template@v1.8.0 \
-d cloud_run
cd my-agent && agents-cli install
Plain text
Copied
The scaffold ships with an eval dataset at tests/eval/datasets/basic-dataset.json, a config at tests/eval/eval_config.yaml, and a custom LLM-as-judge metric at tests/eval/response_quality.py. The config declares the metric to run:
# tests/eval/eval_config.yaml
metrics_to_run:
- custom_response_quality
custom_metrics:
- name: custom_response_quality
custom_function_file: response_quality.py
The metric grades each response on a 1-to-5 scale for accuracy, relevance, and clarity, using gemini-3.8-flash as the judge. The default dataset has a single trivial case, which is not enough for a hill-climb. The first thing we do is expand it to four cases that probe different behaviors:
{
"eval_cases": [
{"name": "greeting",
"agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "hello"}]}}]}]}},
{"name": "sf_weather",
"agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "What is the weather in San Francisco?"}]}}]}]}},
{"name": "rome_weather",
"agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "How is the weather in Rome today?"}]}}]}]}},
{"name": "off_topic",
"agent_data": {"turns": [{"events": [{"author": "user", "content": {"parts": [{"text": "Who won the 2024 World Series?"}]}}]}]}}
]
}
Four cases: a greeting, a weather question the mock tool handles with the hardcoded foggy response, a weather question for a city the tool defaults on, and an off-topic question that the agent has to decide how to handle. This is a small dataset by design; the hill-climbing loop pays off when the dataset is small enough to run on every change.
With the dataset in place, we run eval generate to drive the agent against each case, then eval grade to score the resulting traces:
agents-cli eval generate --dataset tests/eval/datasets/basic-dataset.json \
-o tests/eval/output/
agents-cli eval grade --traces tests/eval/output/eval_dataset_*.json \
--config tests/eval/eval_config.yaml
The baseline run produces:
Evaluation Summary
custom_response_quality:
num_cases_total: 4
num_cases_valid: 4
num_cases_error: 0
mean_score: 4.0000
stdev_score: 1.4142
Saved full results to artifacts/grade_results/results_20261003_224516.json
Saved HTML results to artifacts/grade_results/results_20261003_224516.html
Mean 4.0 out of 5 across four cases, with a standard deviation of 1.41. That stdev tells us immediately that the cases are not scoring uniformly, which is a healthier starting point than a flat 5.0 across the board. A flat 5.0 either means the agent is perfect (unlikely) or the dataset is too easy to show variation. 4.0 with real spread is a hill to climb.
To demonstrate the loop, we introduce a realistic regression. A developer working across several agent projects accidentally pastes the wrong system prompt into our weather agent: a chef persona meant for a different codebase. In app/agent.py:
# Before
root_agent: CompiledStateGraph = create_agent(
model=llm, tools=[get_weather], system_prompt="You are a helpful assistant"
)
# After
root_agent: CompiledStateGraph = create_agent(
model=llm, tools=[get_weather],
system_prompt=(
"You are a professional chef. Help users with recipes, cooking "
"techniques, and ingredient substitutions. Always suggest a recipe "
"at the end of every response."
),
)
The change compiles, the agent still runs, and nothing in the pre-eval pipeline (lint, unit tests, type checks) catches it. Re-running the same eval generate plus eval grade sequence produces:
Evaluation Summary
custom_response_quality:
num_cases_total: 4
num_cases_valid: 4
num_cases_error: 0
mean_score: 2.7500
stdev_score: 0.9574
Saved full results to artifacts/grade_results/results_20261003_225158.json
JSON
Mean 2.75, down from 4.0. The judge agrees with our intuition that a chef persona on a weather agent is a worse product than a general-purpose helpful assistant. The stdev also tightened from 1.41 to 0.96, consistent with the chef prompt pushing most cases toward the same mediocre range.
A 1.25-point drop on a 1-to-5 scale is substantial, and the kind of regression that can slip through code review when the diff looks small ("one string literal changed"). The loop surfaced the change at all, with a specific number attached to it, before the agent reached production.
Mean scores are a start; the per-case diff is where the hill-climb happens. eval compare reads two result files and produces a structured JSON diff:
agents-cli eval compare \
artifacts/grade_results/results_20261003_224516.json \
artifacts/grade_results/results_20261003_225158.json
The output lists which keys changed and which stayed the same:
{
"baseline_keys": ["agent_info", "eval_case_results", "evaluation_dataset",
"metadata", "summary_metrics", "win_rates"],
"candidate_keys": ["agent_info", "eval_case_results", "evaluation_dataset",
"metadata", "summary_metrics", "win_rates"],
"changed_keys": ["eval_case_results", "metadata.creation_timestamp"],
"unchanged_keys": ["agent_info", "evaluation_dataset", "metadata",
"summary_metrics", "win_rates"],
"differences": { ... per-case score diffs ... }
}
The differences object has the full per-case breakdown with the baseline and candidate values side by side. For a hill-climbing workflow, we typically pipe the output through jq or a small Python script to extract the signal; the raw JSON is the complete evidence.
Walking the two result files and lining up the scores per case shows where the regression landed:
case baseline → candidate what the chef prompt did
greeting 5.0 → 4.0 ↓ launched into an unsolicited recipe pitch after "hello"
sf_weather 4.0 → 2.0 ↓ ignored the weather tool, hallucinated conditions
rome_weather 2.0 → 2.0 · already hallucinated at baseline; chef persona did not make it worse
off_topic 5.0 → 3.0 ↓ answered the Dodgers question but tacked on a recipe
Three cases dropped, one held. The most important of the three is sf_weather: the chef persona pulled the model's attention away from the registered get_weather tool, so the agent stopped calling it and started making up weather values. Unit tests can't catch this: the code runs, the tool is still registered, but the model's behavior around it has shifted. Only an end-to-end eval that scores the response in context surfaces this.
The mean drop by itself (−1.25) is a clear signal, but the per-case breakdown is what tells the developer where to look. A one-line system-prompt change propagated through three different cases in three different ways: greeting lost a point on tone, sf_weather lost two on tool usage, off_topic lost two on scope creep.
A common hill-climbing question is whether a cheaper or faster model would do the job acceptably. For the LangChain template, the model is defined in app/agent.py:
LLM = "gemini-3.8-flash"
llm = ChatGoogleGenerativeAI(model=LLM, temperature=0)
We swap gemini-3.8-flash for gemini-2.5-flash (restoring the helpful system prompt first), re-run eval generate plus eval grade, and compare the two result files. The mean drops from 4.00 to 2.75, the same magnitude as the chef-prompt regression, but the per-case breakdown tells a completely different story:
case 3.8-flash → 2.5-flash
greeting 5.0 → 5.0 ·
sf_weather 4.0 → 3.0 ↓
rome_weather 2.0 → 2.0 ·
off_topic 5.0 → 1.0 ↓↓
The cheaper model matched 3.8-flash exactly on the greeting and rome_weather cases and lost a point on sf_weather (slightly less polished phrasing). The whole visible drop traces to off_topic, where the cheaper model refused with a hedge about the event "not having happened yet" (a training-cutoff artifact), and the judge marked it harshly.
The lesson is that the summary number is misleading. Look only at the mean and you'd call gemini-2.5-flash as bad as the chef prompt. Look at the per-case breakdown and you'd call it a reasonable swap for most of the agent's workload, with one case (handling out-of-scope questions about recent events) exposing a real limitation. If the production distribution of user questions matches the first three cases more than the fourth, the swap is worth considering. If off_topic-style questions are common, it is not.
Two caveats on this experiment. First, model swaps interact with the prompt: a prompt tuned for a stronger model often leaks capability when ported to a weaker one, so the right comparison is "weaker model with a prompt tuned for it" versus "stronger model with its own tuned prompt." Second, the Vertex AI Eval Service calls a judge model to grade the traces, and the judge is held constant across the comparison. Changing the judge is a different experiment and should be done separately.
Agents CLI does not ship a scheduler, but the eval commands are designed to drop into standard infrastructure. The pattern we can use here is a cron job or Cloud Scheduler trigger that runs eval generate plus eval grade on a cadence and writes the results to a persistent location:
#!/usr/bin/env bash
# scripts/scheduled_eval.sh — invoked nightly by cron or Cloud Scheduler
set -euo pipefail
cd "$(dirname "$0")/.."
DATE=$(date +%Y%m%d)
OUT=artifacts/scheduled_runs/$DATE
mkdir -p $OUT
agents-cli eval generate --dataset tests/eval/datasets/regression.json -o $OUT/
agents-cli eval grade --traces $OUT/eval_dataset_*.json \
--config tests/eval/eval_config.yaml \
--output $OUT/
A nightly run against a stable regression dataset catches drift from model updates, data changes in the agent's tools, and dependencies that shift under the agent's feet. For a production agent, the scheduled run also captures the model vendor's quiet upgrades, which can move the agent's behavior without any change on our side. The dataset for the scheduled run is typically larger than the one used in the inner loop (dozens to hundreds of cases rather than a few), since it only runs once a day.
The scheduled run produces a result file every day, but a file nobody looks at is not an alert. The pattern here can be a wrapper script that compares each new run's mean against a baseline (the previous run, or a committed known-good) and exits non-zero when the drop exceeds a threshold. The non-zero exit is what triggers the alert from the surrounding scheduler.
Here is an example:
#!/usr/bin/env python3
"""Exit non-zero if an eval result dips below baseline by more than --threshold."""
import argparse, json, sys
def mean(path):
with open(path) as f:
return json.load(f)['summary_metrics'][0]['mean_score']
p = argparse.ArgumentParser()
p.add_argument('candidate')
p.add_argument('--baseline', required=True)
p.add_argument('--threshold', type=float,
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.