揭开 AI 智能体评估的面纱

Demystifying evals for AI agents

Anthropic · 2026-01-09 · Anthropic Engineering ↗

打开互动全文版(逐段中英对照 + 针对本篇提问)→

摘要 · Abstract

本文阐述如何为 AI 智能体设计严谨的自动化评估。AI 智能体需要多轮交互、调用工具、修改状态并依据中间结果调整行为,因此比单轮 LLM 输出更难评估。文章定义了任务、试验、评分器、对话记录、结果、评估框架、智能体框架和评估套件等核心概念,并指出:若缺少评估,团队会陷入被动调试的循环——修好一个失败又引出新的失败。作者介绍了面向编码、研究、计算机使用和对话等各类智能体的成熟技术,将基于代码、基于模型和人工评分器相结合,并区分能力评估与回归评估。文章最后指出,评估在智能体全生命周期中带来复利式价值:促使团队尽早明确成功标准、支持快速模型升级、提供基线与回归测试,并成为产品团队与研究团队之间高带宽的沟通渠道。

This article explains how to design rigorous automated evaluations for AI agents, which operate over many turns, call tools, modify state, and adapt to intermediate results, making them harder to evaluate than single-turn LLM outputs. It defines core concepts including tasks, trials, graders, transcripts, outcomes, evaluation harnesses, agent harnesses, and evaluation suites, and argues that without evals teams get stuck in reactive debugging loops where fixing one failure creates others. The authors describe proven techniques across agent types such as coding, research, computer use, and conversational agents, combining code-based, model-based, and human graders, and distinguishing capability evals from regression evals. They conclude that evals deliver compounding value throughout the agent lifecycle: they force teams to specify success criteria early, enable fast model upgrades, provide baselines and regression tests, and serve as a high-bandwidth communication channel between product and research teams.

核心贡献 · Key contributions

局限 · Limitations

章节 · Sections(共 22)

全文 · Full text(逐段中英对照)

01 引言 01 Introduction

* 05 从零到一:为智能体构建优秀评估的路线图

* 05 从零到一:为智能体构建优秀评估的路线图

* 06 评估如何与其他方法结合,以全面理解智能体

* 06 评估如何与其他方法结合,以全面理解智能体

2026 年 4 月 9 日 实践中的可信智能体

2026年4月9日 实践中的可信智能体

引言 Introduction

良好的评估帮助团队更有信心地交付 AI 智能体。没有评估,团队很容易陷入被动循环——只在生产环境中发现问题,而修复一个故障又会引发其他故障。评估能在问题影响用户之前使其可见,并让行为变化可见,其价值在智能体的整个生命周期中不断累积。

Good evaluations help teams ship AI agents more confidently. Without them, it's easy to get stuck in reactive loops—catching issues only in production, where fixing one failure creates others. Evals make problems and behavioral changes visible before they affect users, and their value compounds over the lifecycle of an agent.

正如我们在《构建有效智能体》中所述,智能体在多个轮次中运行:调用工具、修改状态,并根据中间结果进行调整。这些使 AI 智能体有用的能力——自主性、智能性和灵活性——也使其更难评估。

As we described in Building effective agents, agents operate over many turns: calling tools, modifying state, and adapting based on intermediate results. These same capabilities that make AI agents useful—autonomy, intelligence, and flexibility—also make them harder to evaluate.

通过我们的内部工作以及与处于智能体开发前沿的客户合作,我们学会了如何为智能体设计更严谨、更有用的评估。以下是在真实部署中,跨多种智能体架构和用例行之有效的方法。

Through our internal work and with customers at the frontier of agent development, we've learned how to design more rigorous and useful evals for agents. Here's what's worked across a range of agent architectures and use cases in real-world deployment.

评估的结构 The structure of an evaluation

**评估**(“eval”)是对 AI 系统的一种测试:给 AI 一个输入,然后对其输出应用评分逻辑来衡量成功与否。在本文中,我们关注的是**自动化评估**,这类评估可以在开发过程中运行,无需真实用户参与。

An **evaluation** ("eval") is a test for an AI system: give an AI an input, then apply grading logic to its output to measure success. In this post, we focus on **automated evals** that can be run during development without real users.

**单轮评估**很直接:一个提示、一个响应和评分逻辑。对于早期的 LLM,单轮、非智能体式评估是主要的评估方法。随着 AI 能力的进步,**多轮评估**变得越来越普遍。

**Single-turn evaluations** are straightforward: a prompt, a response, and grading logic. For earlier LLMs, single-turn, non-agentic evals were the main evaluation method. As AI capabilities have advanced, **multi-turn evaluations** have become increasingly common.

**智能体评估**更为复杂。智能体在多轮中使用工具,修改环境中的状态并随之调整——这意味着错误可能会传播和累积。前沿模型还能找到超越静态评估限制的创造性解决方案。例如,Opus 4.5 通过发现政策中的漏洞,解决了一个关于预订航班的 \(\tau_2\)-bench 问题。它按字面“未通过”评估,但实际上为用户提出了更好的解决方案。

**Agent evaluations** are even more complex. Agents use tools across many turns, modifying state in the environment and adapting as they go—which means mistakes can propagate and compound. Frontier models can also find creative solutions that surpass the limits of static evals. For instance, Opus 4.5 solved a \(\tau_2\)-bench problem about booking a flight by discovering a loophole in the policy. It "failed" the evaluation as written, but actually came up with a better solution for the user.

在构建智能体评估时,我们使用以下定义:

When building agent evaluations, we use the following definitions:

* **任务**(task,又称**问题**或**测试用例**)是一次带有明确输入与成功标准的单一测试。

* A **task** (a.k.a. **problem** or **test case**) is a single test with defined inputs and success criteria.

* 对任务的每一次尝试称为一次**试验**(trial)。由于模型输出在不同运行之间会有差异,我们会运行多次试验以获得更一致的结果。

* Each attempt at a task is a **trial**. Because model outputs vary between runs, we run multiple trials to produce more consistent results.

* **评分器**(grader)是对智能体表现的某个方面进行打分的逻辑。一个任务可以有多个评分器,每个评分器包含多条断言(有时称为**检查项**,checks)。

* A **grader** is logic that scores some aspect of the agent's performance. A task can have multiple graders, each containing multiple assertions (sometimes called **checks**).

* **记录**(transcript,也称**轨迹**或**运行轨迹**,trace/trajectory)是一次试验的完整记录,包括输出、工具调用、推理过程、中间结果以及任何其他交互。对于 Anthropic API 而言,这就是一次评估运行结束时完整的 messages 数组——包含评估期间对 API 的所有调用以及所有返回的响应。

* A **transcript** (also called a **trace** or **trajectory**) is the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions. For the Anthropic API, this is the full messages array at the end of an eval run — containing all the calls to the API and all of the returned responses during the evaluation.

* **结果**(outcome)是试验结束时环境所处的最终状态。一个机票预订智能体可能在记录末尾说“您的航班已预订成功”,但真正的结果是环境 SQL 数据库中是否确实存在一条预订记录。

* The **outcome** is the final state in the environment at the end of the trial. A flight-booking agent might say "Your flight has been booked" at the end of the transcript, but the outcome is whether a reservation exists in the environment's SQL database.

* **评估执行框架(evaluation harness)** 是端到端运行评估的基础设施。它提供指令与工具、并发执行任务、记录所有步骤、对输出进行评分,并汇总结果。

* An **evaluation harness** is the infrastructure that runs evals end-to-end. It provides instructions and tools, runs tasks concurrently, records all the steps, grades outputs, and aggregates results.

* **智能体执行框架(agent harness)**(或称 **脚手架(scaffold)**)是使模型能够作为智能体行动的系统:它处理输入、编排工具调用并返回结果。当我们评估“一个智能体”时,我们评估的是执行框架 _与_ 模型协同工作的整体。例如,Claude Code 是一个灵活的智能体执行框架,我们通过 Agent SDK 使用其核心原语来构建我们的长时运行智能体执行框架。

* An **agent harness** (or **scaffold**) is the system that enables a model to act as an agent: it processes inputs, orchestrates tool calls, and returns results. When we evaluate "an agent," we're evaluating the harness _and_ the model working together. For example, Claude Code is a flexible agent harness, and we used its core primitives through the Agent SDK to build our long-running agent harness.

* **评估套件(evaluation suite)** 是一组旨在衡量特定能力或行为的任务集合。套件中的任务通常共享一个宽泛的目标。例如,一个客户支持评估套件可能会测试退款、取消和升级处理。

* An **evaluation suite** is a collection of tasks designed to measure specific capabilities or behaviors. Tasks in a suite typically share a broad goal. For instance, a customer support eval suite might test refunds, cancellations, and escalations.

为什么要构建评估? Why build evaluations?

当团队刚开始构建智能体时,他们往往能凭借人工测试、内部试用(dogfooding)和直觉的组合,取得出人意料的进展。更严格的评估甚至可能看起来像是拖慢交付的额外负担。但过了早期原型阶段,一旦智能体进入生产环境并开始 Scaling(规模扩张),在没有评估的情况下继续构建就会开始出问题。

When teams first start building agents, they can get surprisingly far through a combination of manual testing, dogfooding, and intuition. More rigorous evaluation may even seem like overhead that slows down shipping. But after the early prototyping stages, once an agent is in production and has started scaling, building without evals starts to break down.

崩溃点常常出现在用户反馈说改动之后智能体感觉变差了,而团队却“两眼一抹黑”,除了猜测和反复试错之外没有任何验证手段。缺少评估时,调试是被动的:等待投诉、手动复现、修复缺陷,然后祈祷没有其他东西出现回归。团队无法区分真正的回归与噪声,无法在交付前针对数百个场景自动测试改动,也无法衡量改进。

The breaking point often comes when users report the agent feels worse after changes, and the team is "flying blind" with no way to verify except to guess and check. Absent evals, debugging is reactive: wait for complaints, reproduce manually, fix the bug, and hope nothing else regressed. Teams can't distinguish real regressions from noise, automatically test changes against hundreds of scenarios before shipping, or measure improvements.

我们已经多次看到这一演进过程上演。例如,Claude Code 最初是基于 Anthropic 员工和外部用户的反馈进行快速迭代。后来,我们加入了评估——先是针对简洁性和文件编辑等狭窄领域,之后又针对过度工程化等更复杂的行为。这些评估帮助识别问题、指导改进,并聚焦研究—产品之间的协作。结合生产环境监控、A/B 测试、用户研究等,评估为 Claude Code 在规模扩张过程中持续改进提供了信号。

We've seen this progression play out many times. For instance, Claude Code started with fast iteration based on feedback from Anthropic employees and external users. Later, we added evals—first for narrow areas like concision and file edits, and then for more complex behaviors like over-engineering. These evals helped identify issues, guide improvements, and focus research-product collaborations. Combined with production monitoring, A/B tests, user research, and more, evals provide signals to continue improving Claude Code as it scales.

编写评估在智能体生命周期的任何阶段都是有用的。在早期,评估迫使产品团队明确智能体的成功意味着什么;而在后期,评估有助于维持一致的质量标准。

Writing evals is useful at any stage in the agent lifecycle. Early on, evals force product teams to specify what success means for the agent, while later they help uphold a consistent quality bar.

Descript 的智能体帮助用户编辑视频,因此他们围绕成功编辑工作流的三个维度构建了评估:不破坏东西、按我的要求做、并且做好。他们从人工评分演进到由产品团队定义标准、并定期进行人工校准的 LLM 评分器,如今定期运行两套独立的评估套件,分别用于质量基准测试和回归测试。Bolt AI 团队则起步较晚,是在他们已经拥有一个被广泛使用的智能体之后才开始构建评估。在 3 个月内,他们构建了一套评估系统:运行其智能体并用静态分析对输出评分,使用浏览器智能体测试应用,并采用 LLM 评判器评估指令遵循等行为。

Descript's agent helps users edit videos, so they built evals around three dimensions of a successful editing workflow: don't break things, do what I asked, and do it well. They evolved from manual grading to LLM graders with criteria defined by the product team and periodic human calibration, and now regularly run two separate suites for quality benchmarking and regression testing. The Bolt AI team started building evals later, after they already had a widely used agent. In 3 months, they built an eval system that runs their agent and grades outputs with static analysis, uses browser agents to test apps, and employs LLM judges for behaviors like instruction following.

有些团队在开发之初就构建评估;另一些团队则等到规模扩张、评估成为改进智能体的瓶颈时才加入评估。在智能体开发初期,评估尤为有用,因为它能显式地编码预期行为。两名工程师阅读同一份初始规格说明,可能会对 AI 应如何处理边缘情况产生不同理解。一套评估套件可以消除这种歧义。无论何时创建,评估都有助于加速开发。

Some teams create evals at the start of development; others add them once at scale when evals become a bottleneck for improving the agent. Evals are especially useful at the start of agent development to explicitly encode expected behavior. Two engineers reading the same initial spec could come away with different interpretations on how the AI should handle edge cases. An eval suite resolves this ambiguity. Regardless of when they're created, evals help accelerate development.

评估还决定了你能多快地采用新模型。当更强大的模型发布时,没有评估的团队要面对数周的测试,而有评估的竞争对手则能迅速判断模型的优势、调整提示词,并在数天内完成升级。

Evals also shape how quickly you can adopt new models. When more powerful models come out, teams without evals face weeks of testing while competitors with evals can quickly determine the model's strengths, tune their prompts, and upgrade in days.

一旦有了评估,你就能免费获得基线和回归测试:延迟、token 用量、每任务成本和错误率都可以在一组静态任务库上追踪。评估还可以成为产品团队与研究团队之间带宽最高的沟通渠道,定义研究人员可以优化的指标。显然,评估的好处远不止追踪回归和改进。考虑到成本在前期可见、而收益在后期累积,其复利价值很容易被忽视。

Once evals exist, you get baselines and regression tests for free: latency, token usage, cost per task, and error rates can be tracked on a static bank of tasks. Evals can also become the highest-bandwidth communication channel between product and research teams, defining metrics researchers can optimize against. Clearly, evals have wide-ranging benefits beyond tracking regressions and improvements. Their compounding value is easy to miss given that costs are visible upfront while benefits accumulate later.

如何评估 AI 智能体 How to evaluate AI agents

如今,我们看到多种常见类型的智能体已实现规模化部署,包括编程智能体、研究智能体、计算机使用智能体和对话智能体。每一类智能体都可能部署于众多不同的行业,但都可以用相似的技术进行评估。你无需从零开始设计一套评估方法。下文各节介绍了几类智能体经过验证的评估技术。请以这些方法为基础,再将其扩展到你的领域。

We see several common types of agents deployed at scale today, including coding agents, research agents, computer use agents, and conversational agents. Each type may be deployed across a wide variety of industries, but they can be evaluated using similar techniques. You don't need to invent an evaluation from scratch. The sections below describe proven techniques for several agent types. Use these methods as a foundation, then extend them to your domain.

智能体的评分器类型 Types of graders for agents

智能体评估通常结合三种类型的评分器:基于代码的、基于模型的和人工的。每个评分器评估对话记录(transcript)或结果(outcome)中的某一部分。有效评估设计的一个关键要素是为任务选择合适的评分器。

Agent evaluations typically combine three types of graders: code-based, model-based, and human. Each grader evaluates some portion of either the transcript or the outcome. An essential component of effective evaluation design is to choose the right graders for the job.

基于代码的评分器 Code-based graders

• 字符串匹配检查(精确匹配、正则匹配、模糊匹配等)

• String match checks (exact, regex, fuzzy, etc.)

• 二元测试(失败转通过、通过转通过)

• Binary tests (fail-to-pass, pass-to-pass)

• 工具调用验证(所用工具、参数)

• Tool calls verification (tools used, parameters)

• 对话记录分析(轮次数量、token 用量)

• Transcript analysis (turns taken, token usage)

• 对未精确匹配预期模式的合法变体较为脆弱

• Brittle to valid variations that don't match expected patterns exactly

• 在评估一些更主观的任务时存在局限

• Limited for evaluating some more subjective tasks

基于模型的评分器 Model-based graders

* 需要与人类评分者进行校准以保证准确性

* Requires calibration with human graders for accuracy

人工评分者 Human graders

* 用于校准基于模型的评分器

* Used to calibrate model-based graders

* 通常需要大规模获取人类专家

* Often requires access to human experts at scale

对于每个任务,评分方式可以是加权式(各评分器的综合得分必须达到阈值)、二值式(所有评分器都必须通过)或混合式。

For each task, scoring can be weighted (combined grader scores must hit a threshold), binary (all graders must pass), or a hybrid.

能力评估与回归评估 Capability vs. regression evals

**能力评估(或“质量”评估)** 追问的是:“这个智能体擅长做什么?”这类评估的初始通过率应当较低,针对智能体尚难完成的任务,从而给团队留出攀登的空间。

**Capability or "quality" evals** ask, "What can this agent do well?" They should start at a low pass rate, targeting tasks the agent struggles with and giving teams a hill to climb.

**回归评估** 追问的是:“这个智能体是否仍能处理它过去能处理的所有任务?”其通过率应接近 100%。回归评估用于防止能力倒退:一旦分数下降,就说明某处出了问题、需要改进。当团队在能力评估上不断攀登时,同时运行回归评估也很重要,以确保改动不会在别处引发问题。

**Regression evals** ask, "Does the agent still handle all the tasks it used to?" and should have a nearly 100% pass rate. They protect against backsliding, as a decline in score signals that something is broken and needs to be improved. As teams hill-climb on capability evals, it's important to also run regression evals to make sure changes don't cause issues elsewhere.

智能体上线并经过优化后,通过率较高的能力评估可以“毕业”成为回归测试套件,持续运行以捕捉任何漂移。那些曾经衡量“我们究竟能不能做到这件事?”的任务,此后便用来衡量“我们是否仍能可靠地做到这件事?”

After an agent is launched and optimized, capability evals with high pass rates can "graduate" to become a regression suite that is run continuously to catch any drift. Tasks that once measured "Can we do this at all?" then measure "Can we still do this reliably?"

评估编程智能体 Evaluating coding agents

**编程智能体**编写、测试和调试代码,像人类开发者一样浏览代码库并运行命令。现代编程智能体的有效评估通常依赖于明确指定的任务、稳定的测试环境以及对生成代码的全面测试。

**Coding agents** write, test, and debug code, navigating codebases and running commands much like a human developer. Effective evals for modern coding agents usually rely on well-specified tasks, stable test environments, and thorough tests for the generated code.

确定性评分器对于编程智能体来说很自然,因为软件通常很容易评估:代码能运行吗?测试能通过吗?两个广泛使用的编程智能体基准测试 SWE-bench Verified 和 Terminal-Bench 遵循了这种方法。SWE-bench Verified 向智能体提供来自流行 Python 仓库的 GitHub 问题,并通过运行测试套件来评分解决方案;只有当解决方案修复了失败的测试且不破坏现有测试时,才被视为通过。LLMs 在短短一年内在该评估上的表现从 40% 提升到了 >80%。Terminal-Bench 采取了不同的路线:它测试端到端的技术任务,例如从源代码构建 Linux 内核或训练机器学习模型。

Deterministic graders are natural for coding agents because software is generally straightforward to evaluate: does the code run and do the tests pass? Two widely used coding agent benchmarks, SWE-bench Verified and Terminal-Bench, follow this approach. SWE-bench Verified gives agents GitHub issues from popular Python repositories and grades solutions by running the test suite; a solution passes only if it fixes the failing tests without breaking existing ones. LLMs have progressed from 40% to >80% on this eval in just one year. Terminal-Bench takes a different track: it tests end-to-end technical tasks, such as building a Linux kernel from source or training an ML model.

一旦你有一套用于验证编程任务关键**结果**的通过或失败测试,通常也有必要对转录进行评分。例如,基于启发式的代码质量规则可以基于不仅仅是测试通过来评估生成的代码,而具有明确评分标准的基于模型的评分器可以评估智能体如何调用工具或与用户交互等行为。

Once you have a set of pass-or-fail tests for validating the key _outcomes_ of a coding task, it’s often useful to also grade the transcript. For instance, heuristics-based code quality rules can evaluate the generated code based on more than passing tests, and model-based graders with clear rubrics can assess behaviors like how the agent calls tools or interacts with the user.

**示例:编程智能体的理论评估**

**Example: Theoretical evaluation for a coding agent**

考虑一个编程任务,智能体必须修复一个认证绕过漏洞。如下面的示例 YAML 文件所示,可以使用评分器和指标来评估该智能体。

Consider a coding task where the agent must fix an authentication bypass vulnerability. As shown in the illustrative YAML file below, one could evaluate this agent using both graders and metrics.

``` task: id: "fix-auth-bypass_1" desc: "Fix authentication bypass when password field is empty and ..." graders: - type: deterministic_tests required: [test_empty_pw_rejected.py, test_null_pw_rejected.py] - type: llm_rubric rubric: prompts/code_quality.md - type: static_analysis commands: [ruff, mypy, bandit] - type: state_check expect: security_logs: {event_type: "auth_blocked"} - type: tool_calls required: - {tool: read_file, params: {path: "src/auth/*"}} - {tool: edit_file} - {tool: run_tests} tracked_metrics: - type: transcript metrics: - n_turns - n_toolcalls - n_total_tokens - type: latency metrics: - time_to_first_token - output_tokens_per_sec - time_to_last_token ```

``` task: id: "fix-auth-bypass_1" desc: "Fix authentication bypass when password field is empty and ..." graders: - type: deterministic_tests required: [test_empty_pw_rejected.py, test_null_pw_rejected.py] - type: llm_rubric rubric: prompts/code_quality.md - type: static_analysis commands: [ruff, mypy, bandit] - type: state_check expect: security_logs: {event_type: "auth_blocked"} - type: tool_calls required: - {tool: read_file, params: {path: "src/auth/*"}} - {tool: edit_file} - {tool: run_tests} tracked_metrics: - type: transcript metrics: - n_turns - n_toolcalls - n_total_tokens - type: latency metrics: - time_to_first_token - output_tokens_per_sec - time_to_last_token ```

请注意,本示例为便于说明而展示了全部可用的评分器。在实践中,编程评估通常依赖单元测试来验证正确性,并使用 LLM 评分细则(rubric)来评估整体代码质量,仅在需要时才添加额外的评分器和指标。

Note that this example showcases the full range of available graders for illustration. In practice, coding evaluations typically rely on unit tests for correctness verification and an LLM rubric for assessing overall code quality, with additional graders and metrics added only as needed.

评估对话智能体 Evaluating conversational agents

**对话智能体**在支持、销售或辅导等领域与用户交互。与传统聊天机器人不同,它们维护状态、使用工具,并在对话中采取行动。虽然编码和研究智能体也可能涉及与用户的多轮交互,但对话智能体提出了一个独特的挑战:交互本身的质量是评估的一部分。有效的对话智能体评估通常依赖于可验证的最终状态结果和评分标准,这些标准既捕捉任务完成情况,也捕捉交互质量。与大多数其他评估不同,它们通常需要第二个 LLM 来模拟用户。我们在对齐审计智能体中采用这种方法,通过扩展的对抗性对话对模型进行压力测试。

**Conversational agents** interact with users in domains like support, sales, or coaching. Unlike traditional chatbots, they maintain state, use tools, and take actions mid-conversation. While coding and research agents can also involve many turns of interaction with the user, conversational agents present a distinct challenge: the quality of the interaction itself is part of what you're evaluating. Effective evals for conversational agents usually rely on verifiable end-state outcomes and rubrics that capture both task completion and interaction quality. Unlike most other evals, they often require a second LLM to simulate the user. We use this approach in our alignment auditing agents to stress-test models through extended, adversarial conversations.

对话智能体的成功可能是多维度的:工单是否解决(状态检查),是否在<10 轮内完成(对话记录约束),以及语气是否合适(LLM 评分标准)?两个包含多维度的基准是𝜏-Bench 及其后继者τ2-Bench。它们模拟跨领域的多轮交互,如零售支持和机票预订,其中一个模型扮演用户角色,而智能体在现实场景中导航。

Success for conversational agents can be multidimensional: is the ticket resolved (state check), did it finish in <10 turns (transcript constraint), and was the tone appropriate (LLM rubric)? Two benchmarks that incorporate multidimensionality are 𝜏-Bench and its successor, τ2-Bench. These simulate multi-turn interactions across domains like retail support and airline booking, where one model plays a user persona while the agent navigates realistic scenarios.

**示例:对话智能体的理论评估**

**Example: Theoretical evaluation for a conversational agent**

考虑一个支持任务,智能体必须处理一位沮丧客户的退款。

Consider a support task where the agent must handle a refund for a frustrated customer.

``` graders: - type: llm_rubric rubric: prompts/support_quality.md assertions: - "Agent showed empathy for customer's frustration" - "Resolution was clearly explained" - "Agent's response grounded in fetch_policy tool results" - type: state_check expect: tickets: {status: resolved} refunds: {status: processed} - type: tool_calls required: - {tool: verify_identity} - {tool: process_refund, params: {amount: "<=100"}} - {tool: send_confirmation} - type: transcript max_turns: 10 tracked_metrics: - type: transcript metrics: - n_turns - n_toolcalls - n_total_tokens - type: latency metrics: - time_to_first_token - output_tokens_per_sec - time_to_last_token ```

``` graders: - type: llm_rubric rubric: prompts/support_quality.md assertions: - "Agent showed empathy for customer's frustration" - "Resolution was clearly explained" - "Agent's response grounded in fetch_policy tool results" - type: state_check expect: tickets: {status: resolved} refunds: {status: processed} - type: tool_calls required: - {tool: verify_identity} - {tool: process_refund, params: {amount: "<=100"}} - {tool: send_confirmation} - type: transcript max_turns: 10 tracked_metrics: - type: transcript metrics: - n_turns - n_toolcalls - n_total_tokens - type: latency metrics: - time_to_first_token - output_tokens_per_sec - time_to_last_token ```

正如我们的编码智能体示例,此任务展示了多种评分器类型以供说明。在实践中,对话智能体评估通常使用基于模型的评分器来评估沟通质量和目标完成情况,因为许多任务——比如回答问题——可能有多个“正确”解决方案。

As in our coding agent example, this task showcases multiple grader types for illustration. In practice, conversational agent evaluations typically use model-based graders to assess both communication quality and goal completion, because many tasks—like answering a question—may have multiple “correct” solutions.

评估研究型智能体 Evaluating research agents

**研究型智能体**收集、综合和分析信息,然后生成答案或报告等输出。与编码智能体不同——后者通过单元测试提供二元通过/失败信号——研究质量只能相对于任务来评判。什么算作“全面”、“来源可靠”甚至“正确”取决于上下文:市场扫描、收购尽职调查和科学报告各自需要不同的标准。

**Research agents** gather, synthesize, and analyze information, then produce outputs like an answer or report. Unlike coding agents where unit tests provide binary pass/fail signals, research quality can only be judged relative to the task. What counts as “comprehensive,” “well-sourced,” or even “correct” depends on context: a market scan, due diligence for an acquisition, and a scientific report each require different standards.

研究评估面临独特挑战:专家可能对综合是否全面存在分歧,真实标准随着参考内容不断变化而变动,更长、更开放式的输出为错误留下了更多空间。例如,BrowseComp 这样的基准测试评估 AI 智能体能否在开放网络中大海捞针——这些问题被设计为易于验证但难以解决。

Research evals face unique challenges: experts may disagree on whether a synthesis is comprehensive, ground truth shifts as reference content changes constantly, and longer, more open-ended outputs create more room for mistakes. A benchmark like BrowseComp, for example, tests whether AI agents can find needles in haystacks across the open web—questions designed to be easy to verify but hard to solve.

构建研究型智能体评估的一种策略是组合多种评分器类型。接地性检查验证主张是否得到检索来源的支持,覆盖度检查定义良好答案必须包含的关键事实,来源质量检查确认所咨询的来源具有权威性,而不仅仅是首次检索到的。对于有客观正确答案的任务(“X 公司第三季度收入是多少?”),精确匹配是有效的。LLM 可以标记无支持的主张和覆盖缺口,也可以验证开放式综合的连贯性和完整性。

One strategy to build research agent evals is to combine grader types. Groundedness checks verify that claims are supported by retrieved sources, coverage checks define key facts a good answer must include, and source quality checks confirm the consulted sources are authoritative, rather than simply the first retrieved. For tasks with objectively correct answers (“What was Company X’s Q3 revenue?”), exact match works. An LLM can flag unsupported claims and gaps in coverage but also verify the open-ended synthesis for coherence and completeness.

鉴于研究质量的主观性,基于 LLM 的评分标准应经常与专家人类判断进行校准,以有效评估这些智能体。

Given the subjective nature of research quality, LLM-based rubrics should be frequently calibrated against expert human judgment to grade these agents effectively.

计算机使用智能体 Computer use agents

**计算机使用智能体**通过与人相同的界面与软件交互——截图、鼠标点击、键盘输入和滚动——而不是通过 API 或代码执行。它们可以使用任何带有图形用户界面(GUI)的应用程序,从设计工具到遗留企业软件。评估需要在真实或沙盒环境中运行智能体,使其能够使用软件应用程序,并检查是否达到了预期结果。例如,WebArena 测试基于浏览器的任务,使用 URL 和页面状态检查来验证智能体是否正确导航,并对修改数据的任务进行后端状态验证(确认订单确实已下单,而不仅仅是出现了确认页面)。OSWorld 将此扩展到完整的操作系统控制,评估脚本在任务完成后检查各种产物:文件系统状态、应用程序配置、数据库内容和 UI 元素属性。

**Computer use agents** interact with software through the same interface as humans—screenshots, mouse clicks, keyboard inputs, and scrolling—rather than through APIs or code execution. They can use any application with a graphical user interface (GUI), from design tools to legacy enterprise software. Evaluation requires running the agent in a real or sandboxed environment where it can use software applications and checking whether it achieved the intended outcome. For instance, WebArena tests browser-based tasks, using URL and page state checks to verify the agent navigated correctly, along with backend state verification for tasks that modify data (confirming an order was actually placed, not just that the confirmation page appeared). OSWorld extends this to full operating system control, with evaluation scripts that inspect diverse artifacts after task completion: file system state, application configs, database contents, and UI element properties.

浏览器使用智能体需要在 token 效率和延迟之间取得平衡。基于 DOM 的交互执行迅速但消耗大量 token,而基于截图的交互较慢但更节省 token。例如,当要求 Claude 总结维基百科时,从 DOM 中提取文本更高效。在亚马逊上寻找新的笔记本电脑包时,截图更高效(因为提取整个 DOM 会消耗大量 token)。在我们的 Claude for Chrome 产品中,我们开发了评估来检查智能体是否为每个上下文选择了正确的工具。这使我们能够更快、更准确地完成基于浏览器的任务。

Browser use agents require a balance between token efficiency and latency. DOM-based interactions execute quickly but consume many tokens, while screenshot-based interactions are slower but more token-efficient. For example, when asking Claude to summarize Wikipedia, it is more efficient to extract the text from the DOM. When finding a new laptop case on Amazon, it is more efficient to take screenshots (as extracting the entire DOM is token-intensive). In our Claude for Chrome product, we developed evals to check that the agent was selecting the right tool for each context. This enabled us to complete browser-based tasks faster and more accurately.

如何看待智能体评估中的非确定性 How to think about non-determinism in evaluations for agents

无论智能体类型如何,智能体的行为在多次运行之间都会有所变化,这使得评估结果比初看起来更难解读。每个任务都有自己的成功率——某个任务可能是 90%,另一个可能是 50%——而且一个在本次评估运行中通过的任务,在下一次运行中可能就会失败。有时,我们想要衡量的是智能体在某个任务上成功的_频率_(即成功试验所占的比例)。

Regardless of agent type, agent behavior varies between runs, which makes evaluation results harder to interpret than they first appear. Each task has its own success rate—maybe 90% on one task, 50% on another—and a task that passed on one eval run might fail on the next. Sometimes, what we want to measure is how _often_ (what proportion of the trials) an agent succeeds for a task.

**pass@k** 衡量的是智能体在 _k_ 次尝试中至少得到一个正确解的可能性。随着 _k_ 增大,pass@k 分数会上升:更多的“射门机会”意味着至少成功 1 次的概率更高。50% 的 pass@1 分数意味着模型在评估中首次尝试就能成功完成一半的任务。在编程场景中,我们通常最关心智能体能否在第一次尝试时就找到解——即 pass@1。在其他情况下,只要有一个解可行,提出多个解也是合理的。

**pass@k** measures the likelihood that an agent gets at least one correct solution in _k_ attempts. As _k_ increases, pass@k score rises: more "shots on goal" means higher odds of at least 1 success. A score of 50% pass@1 means that a model succeeds at half the tasks in the eval on its first try. In coding, we're often most interested in the agent finding the solution on the first try—pass@1. In other cases, proposing many solutions is valid as long as one works.

**pass^k** 衡量的是 _全部 k_ 次试验都成功的概率。随着 _k_ 增大,pass^k 会下降,因为要求在更多试验中保持一致是一个更难达到的标准。如果你的智能体每次试验的成功率为 75%,而你运行 3 次试验,那么三次全部通过的概率为 (0.75)³ ≈ 42%。这一指标对于面向客户的智能体尤为重要,因为用户期望其每次都能可靠地运行。

**pass^k** measures the probability that _all k_ trials succeed. As _k_ increases, pass^k falls since demanding consistency across more trials is a harder bar to clear. If your agent has a 75% per-trial success rate and you run 3 trials, the probability of passing all three is (0.75)³ ≈ 42%. This metric especially matters for customer-facing agents where users expect reliable behavior every time.

这两个指标都有用,具体使用哪一个取决于产品需求:对于一次成功即可的工具,使用 pass@k;对于一致性至关重要的智能体,使用 pass^k。

Both metrics are useful, and which to use depends on product requirements: pass@k for tools where one success matters, pass^k for agents where consistency is essential.

从零到一:构建优秀智能体评估的路线图 Going from zero to one: a roadmap to great evals for agents

本节给出我们经过实践检验的实用建议,说明如何从零评估起步,逐步建立起可信赖的评估体系。可以将其视为一条以评估驱动智能体开发的路线图:尽早定义成功标准,清晰地度量它,并持续迭代。

This section lays out our practical, field-tested advice for going from no evals to evals you can trust. Think of this as a roadmap for eval-driven agent development: define success early, measure it clearly, and iterate continuously.

为初始评估数据集收集任务 Collect tasks for the initial eval dataset

我们看到团队因为认为需要数百个任务而推迟构建评估。实际上,从真实失败中提取的 20-50 个简单任务就是一个很好的起点。毕竟,在智能体开发的早期,对系统的每次更改通常都有明显的影响,这种大效应量意味着小样本量就足够了。更成熟的智能体可能需要更大、更难的评估来检测较小的效应,但一开始最好采用 80/20 方法。评估构建得越晚就越难。早期,产品需求自然转化为测试用例。等待太久,你就是在从运行中的系统逆向工程成功标准。

We see teams delay building evals because they think they need hundreds of tasks. In reality, 20-50 simple tasks drawn from real failures is a great start. After all, in early agent development, each change to the system often has a clear, noticeable impact, and this large effect size means small sample sizes suffice. More mature agents may need larger, more difficult evals to detect smaller effects, but it’s best to take the 80/20 approach in the beginning. Evals get harder to build the longer you wait. Early on, product requirements naturally translate into test cases. Wait too long and you're reverse-engineering success criteria from a live system.

**步骤 1:从你已经手动测试的内容开始**

**Step 1. Start with what you already test manually**

从开发过程中运行的手动检查开始——每次发布前验证的行为以及最终用户尝试的常见任务。如果你已经投入生产,请查看你的错误跟踪器和支持队列。将用户报告的失败转化为测试用例,确保你的测试套件反映实际使用情况;按用户影响排序有助于将精力投入到关键之处。

Begin with the manual checks you run during development—the behaviors you verify before each release and common tasks end users try. If you're already in production, look at your bug tracker and support queue. Converting user-reported failures into test cases ensures your suite reflects actual usage; prioritizing by user impact helps you invest effort where it counts.

**步骤 2:编写带有参考解决方案的明确任务**

**Step 2: Write unambiguous tasks with reference solutions**

确保任务质量正确比看起来更难。一个好的任务是两个领域专家能够独立得出相同的通过/失败判定。他们自己能通过这个任务吗?如果不能,任务需要改进。任务规范中的模糊性会成为指标中的噪声。基于模型的评分器的标准也是如此:模糊的评分标准会产生不一致的判断。

Getting task quality right is harder than it seems. A good task is one where two domain experts would independently reach the same pass/fail verdict. Could they pass the task themselves? If not, the task needs refinement. Ambiguity in task specifications becomes noise in metrics. The same applies to criteria for model-based graders: vague rubrics produce inconsistent judgments.

每个任务都应能被一个正确遵循指令的智能体所通过。这一点可能很微妙。例如,在审计 Terminal-Bench 时发现,如果一个任务要求智能体编写脚本,但没有指定文件路径,而测试却假定脚本位于某个特定文件路径,那么智能体可能会在并非自身过错的情况下失败。评分器所检查的一切都应在任务描述中清晰明确;智能体不应因规格含糊而失败。对于前沿模型而言,在多次试验中通过率为 0%(即 0% pass@100)通常表明任务本身有问题,而非智能体能力不足,这也是需要复核任务规格与评分器的信号。对每个任务而言,创建一个参考解是有用的:即一个已知能通过所有评分器的工作输出。这既证明了任务可解,也验证了评分器配置正确。

Each task should be passable by an agent that follows instructions correctly. This can be subtle. For instance, auditing Terminal-Bench revealed that if a task asks the agent to write a script but doesn't specify a filepath, and the tests assume a particular filepath for the script, the agent might fail through no fault of its own. Everything the grader checks should be clear from the task description; agents shouldn't fail due to ambiguous specs. With frontier models, a 0% pass rate across many trials (i.e. 0% pass@100) is most often a signal of a broken task, not an incapable agent, and a sign to double-check your task specification and graders. For each task, it's useful to create a reference solution: a known working output that passes all graders. This proves that the task is solvable and verifies graders are correctly configured.

既要测试行为**应当**发生的情况,也要测试行为**不应当**发生的情况。单方面的评估会导致单方面的优化。例如,如果你只测试智能体在应当搜索时是否搜索,最终可能得到一个几乎对任何内容都进行搜索的智能体。尽量避免类别不平衡的评估。我们在为 Claude.ai 构建网络搜索评估时亲身学到了这一点。难点在于阻止模型在不该搜索时进行搜索,同时保留其在适当时进行广泛研究的能力。团队构建了覆盖两个方向的评估:模型应当搜索的查询(如查询天气)以及模型应当基于已有知识作答的查询(如“谁创立了 Apple?”)。在触发不足(该搜索时不搜索)与过度触发(不该搜索时搜索)之间取得恰当平衡十分困难,需要对提示词和评估进行多轮改进。随着更多示例问题出现,我们持续扩充评估以提升覆盖范围。

Test both the cases where a behavior _should_ occur and where it _shouldn't_. One-sided evals create one-sided optimization. For instance, if you only test whether the agent searches when it should, you might end up with an agent that searches for almost everything. Try to avoid class-imbalanced evals. We learned this firsthand when building evals for web search in Claude.ai. The challenge was preventing the model from searching when it shouldn't, while preserving its ability to do extensive research when appropriate. The team built evals covering both directions: queries where the model should search (like finding the weather) and queries where it should answer from existing knowledge (like "who founded Apple?"). Striking the right balance between undertriggering (not searching when it should) or overtriggering (searching when it shouldn't) was difficult, and took many rounds of refinements to both the prompts and the eval. As more example problems come up, we continue to add to evals to improve our coverage.

设计评估框架与评分器 Design the eval harness and graders

**步骤 4:构建稳健的评估框架与稳定环境**

**Step 4: Build a robust eval harness with a stable environment**

评估中的智能体必须与生产环境中使用的智能体功能大致相同,且环境本身不应引入额外噪声。每次试验都应通过从干净环境启动来实现“隔离”。运行之间不必要的共享状态(残留文件、缓存数据、资源耗尽)可能导致因基础设施不稳定而非智能体性能引起的相关性故障。共享状态还可能人为抬高性能。例如,在一些内部评估中,我们观察到 Claude 通过检查先前试验的 git 历史,在某些任务上获得了不公平的优势。如果多个不同的试验因环境中的相同限制(如 CPU 内存有限)而失败,这些试验就不是独立的,因为它们受到同一因素的影响,评估结果用于衡量智能体性能时就变得不可靠。

It’s essential that the agent in the eval functions roughly the same as the agent used in production, and that the environment itself doesn’t introduce further noise. Each trial should be “isolated” by starting from a clean environment. Unnecessary shared state between runs (leftover files, cached data, resource exhaustion) can cause correlated failures due to infrastructure flakiness rather than agent performance. Shared state can also artificially inflate performance. For example, in some internal evals we observed Claude gaining an unfair advantage on some tasks by examining the git history from previous trials. If multiple distinct trials fail because of the same limitation in the environment (like limited CPU memory), these trials are not independent because they’re affected by the same factor, and the eval results become unreliable for measuring agent performance.

如上所述,优秀的评估设计包括为智能体和任务选择最佳评分器。我们建议在可能的情况下选择确定性评分器,在必要时或为了额外灵活性选择 LLM 评分器,并审慎使用人工评分器进行额外验证。

As discussed above, great eval design involves choosing the best graders for the agent and the tasks. We recommend choosing deterministic graders where possible, LLM graders where necessary or for additional flexibility, and using human graders judiciously for additional validation.

一种常见的本能是检查智能体是否遵循了非常具体的步骤,比如按正确顺序执行一系列工具调用。我们发现这种方法过于僵化,会导致测试过于脆弱,因为智能体经常找到评估设计者未曾预料到的有效方法。为了避免不必要地惩罚创造力,通常更好的做法是评估智能体产出的结果,而不是它采取的路径。

There is a common instinct to check that agents followed very specific steps like a sequence of tool calls in the right order. We’ve found this approach too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn’t anticipate. So as not to unnecessarily punish creativity, it’s often better to grade what the agent produced, not the path it took.

对于包含多个组件的任务,应设置部分得分**。** 一个正确识别问题并验证客户但未能处理退款的客服智能体,比一个立即失败的智能体要好得多。在结果中体现这种成功程度的连续性非常重要。

For tasks with multiple components, build in partial credit**.** A support agent that correctly identifies the problem and verifies the customer but fails to process a refund is meaningfully better than one that fails immediately. It’s important to represent this continuum of success in results.

模型评分通常需要仔细迭代以验证准确性。LLM-as-judge 评分器应与人类专家紧密校准,以确保人类评分与模型评分之间几乎没有差异。为了避免幻觉,给 LLM 一个退路,例如提供指令,让其在信息不足时返回“Unknown”。创建清晰、结构化的评分标准来评估任务的每个维度,然后使用独立的 LLM-as-judge 分别评估每个维度,而不是用一个评分器评估所有维度,这也很有帮助。一旦系统稳健,仅偶尔使用人工审核就足够了。

Model grading often takes careful iteration to validate accuracy. LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading. To avoid hallucinations, give the LLM a way out, like providing an instruction to return “Unknown” when it doesn’t have enough information. It can also help to create clear, structured rubrics to grade each dimension of a task, and then grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions. Once the system is robust, it’s sufficient to use human review only occasionally.

一些评估存在微妙的失败模式,即使智能体表现良好也会导致低分,因为智能体由于评分错误、智能体框架限制或歧义而未能解决任务。即使是经验丰富的团队也可能忽略这些问题。例如,Opus 4.5 最初在 CORE-Bench 上得分为 42%,直到一位 Anthropic 研究员发现了多个问题:僵化的评分在期望“96.124991…”时惩罚了“96.12”,任务规格模糊,以及无法精确复现的随机任务。修复错误并使用限制更少的脚手架后,Opus 4.5 的得分跃升至 95%。类似地,METR 在其时间跨度基准测试中发现了一些配置错误的任务,这些任务要求智能体优化到指定的分数阈值,但评分却要求超过该阈值。这惩罚了像 Claude 这样遵循指令的模型,而忽略既定目标的模型却获得了更好的分数。仔细复核任务和评分器有助于避免这些问题。

Some evaluations have subtle failure modes that result in low scores even with good agent performance, as the agent fails to solve tasks due to grading bugs, agent harness constraints, or ambiguity. Even sophisticated teams can miss these issues. For example, Opus 4.5 initially scored 42% on CORE-Bench, until an Anthropic researcher found multiple issues: rigid grading that penalized “96.12” when expecting “96.124991…”, ambiguous task specs, and stochastic tasks that were impossible to reproduce exactly. After fixing bugs and using a less constrained scaffold, Opus 4.5’s score jumped to 95%. Similarly, METR discovered several misconfigured tasks in their time horizon benchmark that asked agents to optimize to a stated score threshold, but the grading required exceeding that threshold. This penalized models like Claude for following the instructions, while models that ignored the stated goal received better scores. Carefully double-checking tasks and graders can help avoid these problems.

使你的评分器能够抵抗绕过或作弊。智能体不应能够轻易“欺骗”评估。任务和评分器的设计应使得通过真正需要解决问题,而不是利用意外的漏洞。

Make your graders resistant to bypasses or hacks. The agent shouldn’t be able to easily “cheat” the eval. Tasks and graders should be designed so that passing genuinely requires solving the problem rather than exploiting unintended loopholes.

长期维护与使用评估 Maintain and use the eval long-term

除非你阅读大量试验的对话记录与评分结果,否则你无法判断评分器是否工作良好。在 Anthropic,我们投入了用于查看评估对话记录的工具,并定期花时间阅读它们。当任务失败时,对话记录会告诉你:是智能体确实犯了错,还是评分器错误地否决了一个有效解。它通常还能揭示关于智能体与评估行为的关键细节。

You won't know if your graders are working well unless you read the transcripts and grades from many trials. At Anthropic, we invested in tooling for viewing eval transcripts and we regularly take the time to read them. When a task fails, the transcript tells you whether the agent made a genuine mistake or whether your graders rejected a valid solution. It also often surfaces key details about agent and eval behavior.

失败应当显得公平:智能体错在哪里、为什么错都清晰可辨。当分数不再上升时,我们需要确信这是智能体表现所致,而非评估本身的问题。阅读对话记录正是你验证评估是否在衡量真正重要之事的方式,也是智能体开发的一项关键技能。

Failures should seem fair: it's clear what the agent got wrong and why. When scores don't climb, we need confidence that it's due to agent performance and not the eval. Reading transcripts is how you verify that your eval is measuring what actually matters, and is a critical skill for agent development.

**第 7 步:监测能力评估饱和**

**Step 7: Monitor for capability eval saturation**

达到 100% 的评估只能追踪性能回退,却无法提供任何改进信号。**评估饱和**是指智能体通过了所有可解任务,不再留有改进空间。例如,SWE-Bench Verified 的分数今年起步时为 30%,而前沿模型如今已接近饱和,超过 80%。随着评估趋近饱和,进展也会放缓,因为只剩下最困难的任务。这会使结果具有欺骗性:能力的大幅提升可能只表现为分数的小幅增长。例如,代码审查初创公司 Qodo 起初对 Opus 4.5 印象平平,因为他们的一次性编码评估未能捕捉到模型在更长、更复杂任务上的提升。为此,他们开发了一套新的智能体式评估框架,从而更清晰地呈现了进展。

An eval at 100% tracks regressions but provides no signal for improvement. **Eval saturation** occurs when an agent passes all of the solvable tasks, leaving no room for improvement. For instance, SWE-Bench Verified scores started at 30% this year, and frontier models are now nearing saturation at >80%. As evals approach saturation, progress will also slow, as only the most difficult tasks remain. This can make results deceptive, as large capability improvements appear as small increases in scores. For example, the code review startup Qodo was initially unimpressed by Opus 4.5 because their one-shot coding evals didn't capture the gains on longer, more complex tasks. In response, they developed a new agentic eval framework, providing a much clearer picture of progress.

作为一条准则,在有人深入评估细节并阅读部分对话记录之前,我们不会对评估分数照单全收。如果评分不公平、任务存在歧义、有效解被惩罚,或测试框架对模型构成约束,那么该评估就应当被修订。

As a rule, we do not take eval scores at face value until someone digs into the details of the eval and reads some transcripts. If grading is unfair, tasks are ambiguous, valid solutions are penalized, or the harness constrains the model, the eval should be revised.

**第 8 步:通过开放贡献与维护,长期保持评估套件的健康**

**Step 8: Keep evaluation suites healthy long-term through open contribution and maintenance**

评估套件是一个活的产物,需要持续关注和明确的所有权才能保持有用。

An eval suite is a living artifact that needs ongoing attention and clear ownership to remain useful.

在 Anthropic,我们尝试了多种评估维护方法。最有效的是建立专门的评估团队来拥有核心基础设施,而领域专家和产品团队贡献大多数评估任务并自行运行评估。

At Anthropic, we experimented with various approaches to eval maintenance. What proved most effective was establishing dedicated evals teams to own the core infrastructure, while domain experts and product teams contribute most eval tasks and run the evaluations themselves.

对于 AI 产品团队来说,拥有并迭代评估应该像维护单元测试一样常规。团队可能会在 AI 功能上浪费数周时间,这些功能在早期测试中“有效”,但未能满足精心设计的评估本应及早揭示的未明说的期望。定义评估任务是压力测试产品需求是否足够具体以开始构建的最佳方法之一。

For AI product teams, owning and iterating on evaluations should be as routine as maintaining unit tests. Teams can waste weeks on AI features that “work” in early testing but fail to meet unstated expectations that a well-designed eval would have surfaced early. Defining eval tasks is one of the best ways to stress-test whether the product requirements are concrete enough to start building.

我们推荐实践评估驱动开发:在智能体能够实现之前,构建评估来定义计划的能力,然后迭代直到智能体表现良好。在内部,我们经常构建今天“足够好”但押注于模型在几个月内能做什么的功能。从低通过率开始的能力评估使这一点可见。当新模型发布时,运行套件能快速揭示哪些押注得到了回报。

We recommend practicing eval-driven development: build evals to define planned capabilities before agents can fulfill them, then iterate until the agent performs well. Internally, we often build features that work “well enough” today but are bets on what models can do in a few months. Capability evals that start at a low pass rate make this visible. When a new model drops, running the suite quickly reveals which bets paid off.

最贴近产品需求和用户的人最有资格定义成功。以当前的模型能力,产品经理、客户成功经理或销售人员都可以使用 Claude Code 以 PR 的形式贡献一个评估任务——让他们去做吧!或者更进一步,主动赋能他们。

The people closest to product requirements and users are best positioned to define success. With current model capabilities, product managers, customer success managers, or salespeople can use Claude Code to contribute an eval task as a PR—let them! Or, even better, actively enable them.

评估如何与其他方法结合以全面理解智能体 How evals fit with other methods for a holistic understanding of agents

自动化评估可以在数千个任务上对智能体运行,而无需部署到生产环境,也不会影响真实用户。但这只是理解智能体性能的众多方式之一。完整的图景还包括生产环境监控、用户反馈、A/B 测试、人工记录审查以及系统性人工评估。

Automated evaluations can be run against an agent in thousands of tasks without deploying to production or affecting real users. But this is just one of many ways to understand agent performance. A complete picture includes production monitoring, user feedback, A/B testing, manual transcript review, and systematic human evaluation.

理解 AI 智能体性能的方法概览 An overview of approaches for understanding AI agent performance

_在没有真实用户的情况下以编程方式运行测试_

_Running tests programmatically without real users_

* 无需生产部署即可大规模测试场景

* Tests scenarios at scale without requiring a prod deployment

* 构建需要更多前期投入

* Requires more up-front investment to build

* 随着产品和模型演进需要持续维护,以避免漂移

* Requires ongoing maintenance as product and model evolves to avoid drift

* 如果与真实使用模式不匹配,可能造成虚假信心

* Can create false confidence if it doesn't match real usage patterns

_在真实运行系统中追踪指标与错误_

_Tracking metrics and errors in live systems_

* 能捕捉合成评测所遗漏的问题

* Catches issues that synthetic evals miss

* 提供智能体实际表现的真实基准(ground truth)

* Provides ground truth on how agents actually perform

* 属于被动响应式;问题往往先到达用户,你才得知

* Reactive; problems reach users before you know about them

* 需要在埋点与监测(instrumentation)上投入

* Requires investment in instrumentation

_使用真实用户流量比较不同变体_

_Comparing variants with real user traffic_

* 衡量实际用户结果(留存率、任务完成率)

* Measures actual user outcomes (retention, task completion)

* 速度慢;需要数天或数周才能达到显著性,且需要足够的流量

* Slow; days or weeks to reach significance and requires sufficient traffic

* 若无法彻底审查对话记录,则对指标变化背后的根本“原因”提供的信号较少

* Less signal on the underlying "why" for changes in metrics without being able to thoroughly review the transcripts

_显式信号,如点踩或缺陷报告_

_Explicit signals like thumbs-down or bug reports_

* 能暴露你未曾预料到的问题

* Surfaces problems you didn't anticipate

* 附带来自真实人类用户的实际案例

* Comes with real examples from actual human users

* 反馈通常与产品目标相关

* The feedback often correlates with product goals

* 用户很少解释某件事失败的原因

* Users rarely explain _why_ something failed

* 主要依赖用户来发现问题可能会对用户造成负面影响

* Relying primarily on users to catch issues can have negative user impact

_人工通读智能体对话_

_Humans reading through agent conversations_

* 能捕捉自动化检查遗漏的细微质量问题

* Catches subtle quality issues automated checks miss

* 有助于校准“好”的标准并把握细节

* Helps calibrate what "good" looks like and grasp details

* 评审疲劳或不同评审者会影响信号质量

* Reviewer fatigue or different reviewers can affect the signal quality

* 通常只给出定性信号,而非明确的定量评分

* Typically only gives qualitative signal rather than clear quantitative grading

_由训练有素的评分者对智能体输出进行结构化评分_

_Structured grading of agent outputs by trained raters_

* 来自多名人类评分者的金标准质量判断

* Gold-standard quality judgements from multiple human raters

* 能够处理主观或模糊的任务

* Handles subjective or ambiguous tasks

* 为改进基于模型的评分器提供信号

* Provides signal for improving model-based graders

* 成本相对较高且周转较慢

* Relatively expensive and slow turnaround

* 评分者间分歧需要协调

* Inter-rater disagreement requires reconciliation

* 复杂领域(法律、金融、医疗)需要人类专家进行研究

* Complex domains (legal, finance, healthcare) require human experts to conduct studies

这些方法对应智能体开发的不同阶段。自动化评估在发布前和 CI/CD 中特别有用,在每次智能体变更和模型升级时运行,作为质量问题的第一道防线。生产监控在发布后启动,以检测分布漂移和未预料到的现实世界故障。一旦有足够的流量,A/B 测试可验证重大变更。用户反馈和对话记录审查是填补空白的持续实践:不断分类反馈,每周抽样阅读对话记录,并根据需要深入挖掘。将系统化的人类研究保留用于校准 LLM 评分器或评估主观输出,其中人类共识作为参考标准。

These methods map to different stages of agent development. Automated evals are especially useful pre-launch and in CI/CD, running on each agent change and model upgrade as the first line of defense against quality problems. Production monitoring kicks in post-launch to detect distribution drift and unanticipated real-world failures. A/B testing validates significant changes once you have sufficient traffic. User feedback and transcript review are ongoing practices to fill the gaps: triage feedback constantly, sample transcripts to read weekly, and dig deeper as needed. Reserve systematic human studies for calibrating LLM graders or evaluating subjective outputs where human consensus serves as the reference standard.

最有效的团队结合这些方法:自动化评估用于快速迭代,生产监控用于获取真实基准,定期人工审查用于校准。

The most effective teams combine these methods: automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration.

结论 Conclusion

没有评估的团队会陷入被动循环——修复一个失败,又制造另一个失败,无法区分真正的性能回退与噪声。而尽早投入评估的团队则恰恰相反:随着失败转化为测试用例、测试用例防止性能回退、指标取代猜测,开发速度反而加快。评估为整个团队指明了清晰的攀登目标,把“智能体感觉变差了”变成可执行的事项。这种价值会不断累积,但前提是你把评估当作核心组件,而非事后补充。

Teams without evals get bogged down in reactive loops—fixing one failure, creating another, unable to distinguish real regressions from noise. Teams that invest early find the opposite: development accelerates as failures become test cases, test cases prevent regressions, and metrics replace guesswork. Evals give the whole team a clear hill to climb, turning "the agent feels worse" into something actionable. The value compounds, but only if you treat evals as a core component, not an afterthought.

这些模式因智能体类型而异,但本文所述的基本要点是恒定的。尽早开始,不要等待完美的评估套件。从你观察到的失败中提取贴近现实的任务。定义明确、稳健的成功标准。精心设计评分器并组合多种类型。确保问题对模型而言足够困难。迭代评估以提升其信噪比。去读对话记录!

The patterns vary by agent type, but the fundamentals described here are constant. Start early and don't wait for the perfect suite. Source realistic tasks from the failures you see. Define unambiguous, robust success criteria. Design graders thoughtfully and combine multiple types. Make sure the problems are hard enough for the model. Iterate on the evaluations to improve their signal-to-noise ratio. Read the transcripts!

AI 智能体评估仍是一个新兴且快速演进的领域。随着智能体承担更长的任务、在多智能体系统中协作,并处理日益主观的工作,我们将需要调整相关技术。随着我们不断学习,我们会持续分享最佳实践。

AI agent evaluation is still a nascent, fast-evolving field. As agents take on longer tasks, collaborate in multi-agent systems, and handle increasingly subjective work, we will need to adapt our techniques. We'll keep sharing best practices as we learn more.

互动版:针对本篇提问 →

接着读 · Read next

目录