由费利克·陈(Felix Chen)，斯德芬·拉班瑟(Stephan Rabanser)，萨亚什·卡普尔(Sayash Kapoor)和阿尔文德·纳拉亚南(Arvind Narayanan)出演。 参见本文完整的作者列表和隶属关系。
越来越多的人呼吁独立和嵌入式AI评价，特别是对AI agents的评价。 但是，许多情况仍然不清楚：什么是“好”评价，首先需要评价什么，以及如何最好地协调和支持大批进入新兴评价生态系统的人。
我们普林斯顿的小组三年来一直在努力改善AI agent的评价状况。 我们与在斯坦福、英国AI安全研究所和布鲁金斯等地做过基础评估工作的人士合作。 我们的文件题为“AI Agent评价中的新趋势”。 我们综合了我们从三个方面——科学与方法、研究议程和评价生态系统——看到的趋势，目的是帮助那些正在提高这种新模式技能的人。 我们还旨在为评估者、资助者和公众提供一套问题，以便提出可以采取行动的建议。
agents，新旧的挑战
AI开发人员长期以来根据基准测试分数，将声称制作到他们系统真实世界的表现。 现在更难评估所收集的证据是否支持声称对agents所做的。 评价人员必须仔细审查整个轨迹，而不仅仅是最后结果。 例如，重要的是要知道agents是否采取捷径，特别是因为他们可以使用各种工具和环境。 而当agents的性能好或坏时，将这些变化归为agent的不同组件就更难了。 解决这些关切需要大量工程努力，而且这项工作越来越昂贵——例如，勘探本克基准测试上的几个agents花费了数以万计的信贷来评估。 这些数字也不计入建立和维护评价基础设施本身所需的大量工程工作。
面对agents产生的大量痕迹，评估人员也在努力进行大规模检查和监测这些agents. 今天依靠工具评估人员来完成这项工作并非无庸置疑，他们自己也需要评估。 为了调查最近发生的OpenAI-HuggingFace事件，评估人员依靠其他agents寻找和查明有害行动的情况，他们承认这种做法有其自身的局限性和偏见。
由于agents被部署在动态环境中(并越来越多地与其他agents一起)，因此更难设计出良好的度量衡来捕捉所关注的基本概念，并为执行这些概念创造出有效的条件。 应对这些挑战需要更好的工具，更严格的基准测试，以及采用不同方法和格式的实验，例如通过开放世界评价和不断增长的关于现实世界使用情况的观测数据。
评价需求的范围正在扩大
关键的是，不能保证一个范围的评价结果转化为另一个范围。 agent可能在实验室中展示出能力，但在特定使用的情况下会面临限制。 考虑METR的随机控制试验，发现尽管有预期，Agents也许能够执行某些任务，但这种能力如何在现实世界中对人类进行翻译，需要根据具体情况进行评价。 因此，需要扩大研究议程，以便了解agents的每个级别：它们的能力、它们与人类的互动以及对机构和激励结构的影响。 这项工作要求技术和社会技术评估不仅要研究agent本身，还要研究agent(s)的背景。
在这些不同的层次上，评价人员可以选择设计他们提出的问题和产生的结果。 agent能否实现任务或处理自动化？或者，agent能增强人类的工作吗？这是两个互补的镜头。 如果评价主要侧重于agents如何将某些任务自动化，我们将更清楚地了解在哪里以及如何可以取代人类工作。 agents如何改善人类工作。
谁来评估？
这一扩大的研究议程将需要从多个学科中扩大一组评价人员。 它还要求评估人员有选择应处理哪些问题以及如何解决的自由。 如果采用不同方法的评估人员同意，我们可能会对这些结果获得更大的信心。 而当冲突发生时，评价人员可以找出模糊的定义和对其结论的限制。
谁来参与，以及以何种独立性，将塑造评估人员提出的问题和他们使用的方法。 有意义的独立性受到若干因素的影响。 随着前沿评价费用的不断增加，独立评价员的参与能力将越来越多地由他们获得更多资金来决定。 独立评估人员能否充分接触模型，也影响到他们衡量和评估真正前沿的能力。 进入问题现在也扩大到agents的 " 思想链 " 痕迹，因为前沿实验室现在发布摘要版本而不是原始痕迹。 除了这些挑战外，评估人员还需要了解他们是否能够在不担心报复的情况下开展工作，以及如何透明地分享他们的调查结果。
我们从何说起？
现在有一个明确的窗口，为agent评价带来新的参与者，方法和想法，并满足快速移动空间对严格证据的迫切需要。 我们希望这篇论文让读者对实地的方向有了新的认识。 AI评价界的利益攸关方可以采取许多步骤来推进AI评价的科学和实践。
对于未来的评估人员来说，需要确定在这一生态系统中哪些作用或问题最适合他们的专门知识，并审查这项工作需要哪些层面的独立性。 特别是，我们鼓励尚未参与大赦国际评估的学科参与；有许多公开的挑战可以从它们的观点中受益。
对目前的评价人员来说，有机会审查他们自己的方法假设并接受其他学科的观点；agent评价的这个领域本质上是跨学科的，必然涉及对现有想法和见解的重新发现和再造。
对于实地建设者——资助者、联盟和会议组织者——agent评价需要自己的实践领域，一个具有更高有效性和严格标准的实践领域。 通过汇集最佳做法，促进不同纪律观点之间的对话和接触，可以提高这一条。
对于领导板聚合者来说，他们通过确定评价的哪些方面受到重视，以及以何种方式确定评价的哪些方面，是这一新出现的生态系统的核心，因此，有极好的机会将最佳做法纳入下游采用的报告，并帮助公众了解评价成本的上升。
就部署人员和行业评价人员而言，现实世界环境在为评价工作提供信息方面存在重大差距。 通过在现场提供真知灼见并参与实验室的工作，他们可以加强这个快速移动空间中的对话。
对于实验室来说，有机会与独立的评估人员建立桥梁，并审查社区的最佳做法。 成功的嵌入式评价人员需要通过独立性、透明度和方法的严格性获得公众合法性。
对决策者来说，需要形成对当前生态系统的明确认识，以及良好的评价应该是什么样子。 对于公共部门的部署，评价必须从第一天起就成为该进程的一个综合和持续的组成部分。 AISI可以发挥关键作用，开创新的合作模式，并邀请新的利益攸关方加入其在当前生态系统内的公共投入机制。
在这里看报纸
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
By Felix Chen, Stephan Rabanser, Sayash Kapoor, and Arvind Narayanan. See the full author list and affiliations here.
There are growing calls for independent and embedded AI evaluation, especially of AI agents. But much remains unclear: what constitutes a “good” evaluation, what needs to be evaluated in the first place, and how best to coordinate and support the influx of people entering the burgeoning evaluation ecosystem.
Our group at Princeton has been working to improve the state of AI agent evaluation for about three years. We teamed up with people who’ve done foundational work on evaluation at Stanford, the UK AI Security Institute, and Brookings. Our paper is titled Emerging Trends in AI Agent Evaluation. We synthesized the trends we see across three dimensions—the science and methods, the research agenda, and the evaluation ecosystem—with the goal to help those who are upskilling for this new paradigm. We also aim to equip evaluators, funders, and the public with a set of questions to ask, and provide actionable recommendations.
Challenges for agents, new and old
AI developers have long made claims about the real-world performance of their systems based on benchmark scores. It is harder now to assess whether the evidence gathered supports claims made about agents. The evaluator must scrutinize the entire trajectory, not just the final outcome. It is important, for example, to know if agents took shortcuts, particularly since they have access to tools and environments. And when agents perform better or worse, it is harder to attribute these changes to different components of the agent. Addressing these concerns requires significant engineering effort, and the work is getting increasingly expensive—for example, several agents on the ExploitBench benchmark cost tens of thousands of dollars in credits to evaluate. These numbers also don’t factor in the significant engineering effort needed to set up and maintain the evaluation infrastructure itself.
Faced with the massive volume of traces that agents generate, evaluators are also struggling to inspect and monitor these agents at scale. The tools evaluators rely on today to do this work are not foolproof—and themselves require evaluation. To investigate the recent OpenAI-HuggingFace incident after the fact, evaluators relied upon other agents to look for and identify instances of harmful action, an approach they acknowledged carried its own limitations and biases.
Because agents are deployed in dynamic settings (and increasingly together with other agents), it is harder to design good metrics to capture the underlying concepts of interest, and create valid conditions to implement them. Addressing these challenges will require better tools, more rigorous benchmarks, and experimentation with different approaches and formats, such as through open-world evaluations and growing observational data on real-world usage.
The scope of evaluation needs is growing
Crucially, the results of evaluation at one level of scope are not guaranteed to translate to another. An agent might demonstrate capability in the lab, but face limits when placed in a specific use case. Consider METR’s randomized controlled trial which found that, despite expectations, usage of agent tools toward software development actually increased overall completion time on the tasks at hand. Agents may be able to perform certain tasks, but how that capability translates for humans in the real world requires evaluation in context. An expanded research agenda is therefore needed to understand agents at each level: their capabilities, their interactions with humans, and their impact on institutions and incentive structures. This work requires both technical and sociotechnical evaluation to study not just the agent itself but the agent(s) in context.
Across these different layers, evaluators have a choice in framing the questions they ask and the findings they generate. Can an agent automate a task or process? Or, can an agent augment the work of a human? These are two complementary lenses. If evaluation largely focuses on how agents automate certain tasks, we will better understand where and how human work can be replaced. But it won’t tell us how humans can better collaborate using agents, or how agents can improve human work.
Who gets to evaluate and how?
This expanded research agenda will require an expanded set of evaluators from a mix of disciplines. It will also require that evaluators have the freedom to choose which questions to pursue and how. When evaluators with different approaches agree, we might gain greater confidence in those results. And when they conflict, evaluators can identify vague definitions and limitations to their findings.
Who gets to participate, and with what independence, will shape the questions evaluators ask and the methods they use. Meaningful independence is influenced by several factors. As the costs of frontier evaluation keep increasing, the ability of independent evaluators to participate will be increasingly determined by their access to increased funding. Whether independent evaluators have sufficient access to models also impacts their ability to measure and assess the true frontier. Questions of access now also extend to Chain of Thought traces for agents, since frontier labs now release summarized versions instead of raw traces. Alongside these challenges, evaluators also need to navigate whether they can conduct work without fear of retaliation, and how they can transparently share their findings.
Where do we go from here?
There is a clear window now to bring in new participants, methods, and ideas for agent evaluations, and to meet the pressing need for rigorous evidence in a fast-moving space. We hope this paper gives readers an emerging sense of the field’s direction. There are many steps that stakeholders in the AI evaluation community can take to advance the science and practice of AI evaluation.
For prospective evaluators, there is a need to determine what roles or questions within this ecosystem best fit their expertise, and examine which dimensions of independence are required for that work. In particular, we encourage participation from disciplines not yet engaged in AI evaluation; there are many open challenges that can benefit from their perspectives.
For current evaluators, there is an opportunity to examine their own methodological assumptions and embrace perspectives from other disciplines; this field of agent evaluation is inherently interdisciplinary, and will necessarily involve some rediscovery and reinvention of existing ideas and insights.
For field-builders—funders, coalitions, and conference organizers—agent evaluation demands its own field of practice, one with higher standards for validity and rigor. Raising this bar can happen by coalescing best practices and fostering conversation and engagement between different disciplinary perspectives.
For leaderboard aggregators, who are central to this emerging ecosystem by shaping which dimensions of evaluation receive attention and in what manner, there is a prime opportunity to incorporate best practices around reporting for downstream adoption, and to help the public understand the rising costs of evaluation.
For deployers and industry evaluators, there is a critical gap for real-world settings to inform evaluation work. By contributing on-the-ground insights and engaging with work in the lab, they can strengthen the discourse in this fast-moving space.
For labs, there is an opportunity to build bridges with independent evaluators and examine the community’s best practices. Successful embedded evaluators will require public legitimacy through independence, transparency, and methodological rigor.
For policymakers, there is a need to develop a clear sense of the current ecosystem and what good evaluations should look like. And for public sector deployments, evaluation needs to be an integrated and continuous part of the process from day one. AISIs can play a key role by pioneering new models for collaboration and inviting new stakeholders into their public input mechanisms within the current ecosystem.
Read the paper here.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.