面向长时间运行应用的多智能体框架设计

Harness design for long-running apps \ Anthropic

Anthropic · 2026-03-24 · Anthropic Engineering ↗

打开互动全文版(逐段中英对照 + 针对本篇提问)→

摘要 · Abstract

本文介绍了一种面向长时间自主编码与主观前端设计的多智能体框架设计,其灵感来自生成对抗网络。作者指出,朴素实现中存在两种顽固的失败模式:随着上下文窗口被填满,模型会失去连贯性或表现出“上下文焦虑”;同时,智能体总是倾向于赞美自己的作品,而非对其进行批判性评估。为解决这些问题,该框架将生成智能体与评估智能体分离,并将评估智能体调校为持怀疑态度,同时为其提供具体的评分标准,把主观判断转化为可评分的指标。在前端设计方面,设计质量、原创性、工艺和功能性四项标准被赋予偏向设计与原创性的权重,以推动生成器超越千篇一律的输出;经过五到十五轮迭代反馈循环,结果会愈发独特。在全栈开发方面,由规划者、生成者和评估者组成的三智能体架构可将简短提示扩展为产品规格,并在长达数小时的会话中构建出丰富的应用。结论是:将生成与评估分离,并结合结构化标准与上下文管理,能够显著提升主观设计质量和可验证的软件成果。

This article describes a multi-agent harness design for long-running autonomous coding and subjective frontend design, drawing inspiration from generative adversarial networks. The author identifies two persistent failure modes in naive implementations: models lose coherence or exhibit context anxiety as the context window fills, and agents reliably praise their own work rather than evaluating it critically. To address these, the harness separates a generator agent from an evaluator agent, with the evaluator tuned to be skeptical and given concrete grading criteria that translate subjective judgments into gradable terms. For frontend design, four criteria—design quality, originality, craft, and functionality—are weighted toward design and originality to push the generator beyond generic outputs, and an iterative feedback loop of five to fifteen cycles produces increasingly distinctive results. For full-stack development, a three-agent architecture of planner, generator, and evaluator expands a short prompt into a product spec and builds rich applications over multi-hour sessions. The conclusion is that separating generation from evaluation, combined with structured criteria and context management, substantially improves both subjective design quality and verifiable software outcomes.

核心贡献 · Key contributions

局限 · Limitations

章节 · Sections(共 10)

全文 · Full text(逐段中英对照)

面向长时间运行应用开发的 Harness 设计 Harness design for long-running application development

* 01 为什么朴素实现会失败

* 01 为什么朴素实现会失败

* 02 前端设计:让主观质量可分级

* 02 前端设计:让主观质量可分级

_作者:Prithvi Rajasekaran,我们 Labs 团队成员。_

_作者:Prithvi Rajasekaran,我们 Labs 团队成员。_

2026 年 2 月 5 日 用一组并行 Claude 构建 C 编译器

2026年2月5日 用一组并行 Claude 构建 C 编译器

过去几个月里,我一直致力于两个相互关联的问题:让 Claude 产出高质量的前端设计,以及让它无需人工干预就能构建完整的应用程序。这项工作源于我们早先在前端设计技能和长时间运行编码智能体 harness 上的努力,我和同事通过提示工程和 harness 设计将 Claude 的性能提升到远高于基线水平——但两者最终都遇到了瓶颈。

过去几个月里,我一直致力于两个相互关联的问题:让 Claude 产出高质量的前端设计,以及让它无需人工干预就能构建完整的应用程序。这项工作源于我们早先在前端设计技能和长时间运行编码智能体 harness 上的努力,我和同事通过提示工程和 harness 设计将 Claude 的性能提升到远高于基线水平——但两者最终都遇到了瓶颈。

为了取得突破,我寻找了一些新颖的 AI 工程方法,这些方法在两个截然不同的领域中都能成立:一个由主观品味定义,另一个由可验证的正确性和可用性定义。受生成对抗网络(GAN)的启发,我设计了一种包含 **生成器(generator)** 与 **评估器(evaluator)** 智能体的多智能体结构。要构建一个能够可靠且带有品味地给输出打分的评估器,首先需要开发一套标准,把诸如“这个设计好吗?”之类的主观判断转化为具体、可评分的条目。

To break through, I sought out novel AI engineering approaches that held across two quite different domains, one defined by subjective taste, the other by verifiable correctness and usability. Taking inspiration from Generative Adversarial Networks (GANs), I designed a multi-agent structure with a **generator** and **evaluator** agent. Building an evaluator that graded outputs reliably—and with taste—meant first developing a set of criteria that could turn subjective judgments like "is this design good?" into concrete, gradable terms.

随后我将这些技术应用于长时间运行的自主编码,并沿用了我们此前框架工作中的两条经验:将构建过程分解为可处理的块,以及使用结构化产物在会话之间交接上下文。最终结果是一种三智能体架构——规划器、生成器和评估器——它能在长达数小时的自主编码会话中产出丰富的全栈应用。

I then applied these techniques to long-running autonomous coding, carrying over two lessons from our earlier harness work: decomposing the build into tractable chunks, and using structured artifacts to hand off context between sessions. The final result was a three-agent architecture—planner, generator, and evaluator—that produced rich full-stack applications over multi-hour autonomous coding sessions.

为何朴素实现难以奏效 Why naive implementations fall short

我们之前已经展示过,harness 设计对长时间运行的智能体式编码的有效性有重大影响。在早期实验中,我们使用初始化智能体将产品规格分解为任务列表,并使用编码智能体一次实现一个功能,然后交接工件以跨会话携带上下文。更广泛的开发者社区已经得出了类似的见解,诸如“Ralph Wiggum”方法使用钩子或脚本让智能体保持连续迭代循环。

We've previously shown that harness design has a substantial impact on the effectiveness of long-running agentic coding. In an earlier experiment, we used an initializer agent to decompose a product spec into a task list, and a coding agent that implemented the tasks one feature at a time before handing off artifacts to carry context across sessions. The broader developer community has converged on similar insights, with approaches like the "Ralph Wiggum" method using hooks or scripts to keep agents in continuous iteration cycles.

但一些问题仍然存在。对于更复杂的任务,智能体仍然倾向于随着时间的推移而偏离轨道。在分解这个问题时,我们观察到智能体执行这类任务时有两种常见的失败模式。

But some problems remained persistent. For more complex tasks, the agent still tends to go off the rails over time. While decomposing this issue, we observed two common failure modes with agents executing these sorts of tasks.

首先,随着上下文窗口被填满,模型在冗长任务上往往会失去连贯性(参见我们关于上下文工程的文章)。一些模型还表现出“上下文焦虑”,即当它们接近自认为的上下文限制时,会过早地开始收尾工作。上下文重置——完全清空上下文窗口并启动一个新的智能体,结合携带前一个智能体状态和后续步骤的结构化交接——解决了这两个问题。

First is that models tend to lose coherence on lengthy tasks as the context window fills (see our post on context engineering). Some models also exhibit "context anxiety," in which they begin wrapping up work prematurely as they approach what they believe is their context limit. Context resets—clearing the context window entirely and starting a fresh agent, combined with a structured handoff that carries the previous agent's state and the next steps—addresses both these issues.

这与压缩不同,压缩是在原地总结对话的早期部分,以便同一个智能体可以在缩短的历史记录上继续工作。虽然压缩保持了连续性,但它没有给智能体一个干净的状态,这意味着上下文焦虑仍然可能存在。重置提供了一个干净的状态,代价是交接工件需要包含足够的状态,以便下一个智能体能够干净地接手工作。在我们早期的测试中,我们发现 Claude Sonnet 4.5 表现出的上下文焦虑足够强烈,以至于仅靠压缩不足以实现强大的长任务性能,因此上下文重置成为 harness 设计的关键。这解决了核心问题,但增加了每次 harness 运行的编排复杂性、token 开销和延迟。

This differs from compaction, where earlier parts of the conversation are summarized in place so the same agent can keep going on a shortened history. While compaction preserves continuity, it doesn't give the agent a clean slate, which means context anxiety can still persist. A reset provides a clean slate, at the cost of the handoff artifact having enough state for the next agent to pick up the work cleanly. In our earlier testing, we found Claude Sonnet 4.5 exhibited context anxiety strongly enough that compaction alone wasn't sufficient to enable strong long task performance, so context resets became essential to the harness design. This solves the core issue, but adds orchestration complexity, token overhead, and latency to each harness run.

第二个问题,我们之前没有解决,是自我评估。当被要求评估它们所产生的工作时,智能体倾向于自信地赞扬工作——即使对人类观察者来说,质量明显平庸。这个问题在像设计这样的主观任务中尤为明显,因为没有等同于可验证软件测试的二元检查。布局感觉是精致还是通用是一种主观判断,而智能体在给自己的作品评分时总是倾向于正面。

A second issue, which we haven't previously addressed, is self-evaluation. When asked to evaluate work they've produced, agents tend to respond by confidently praising the work—even when, to a human observer, the quality is obviously mediocre. This problem is particularly pronounced for subjective tasks like design, where there is no binary check equivalent to a verifiable software test. Whether a layout feels polished or generic is a judgment call, and agents reliably skew positive when grading their own work.

然而,即便在确实具有可验证结果的任务上,智能体有时仍会表现出糟糕的判断力,从而妨碍其完成任务时的表现。将执行工作的智能体与评判工作的智能体分离,被证明是解决这一问题的有力手段。这种分离本身并不能立即消除那种宽容倾向;评估器仍然是一个 LLM,倾向于对 LLM 生成的输出给予宽容。但将独立的评估器调校为持怀疑态度,远比让生成器对自身工作持批判态度更为可行;而且一旦这种外部反馈存在,生成器就有了具体的依据来进行迭代。

However, even on tasks that do have verifiable outcomes, agents still sometimes exhibit poor judgment that impedes their performance while completing the task. Separating the agent doing the work from the agent judging it proves to be a strong lever to address this issue. The separation doesn't immediately eliminate that leniency on its own; the evaluator is still an LLM that is inclined to be generous towards LLM-generated outputs. But tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work, and once that external feedback exists, the generator has something concrete to iterate against.

前端设计:让主观质量可评分 Frontend design: making subjective quality gradable

我从前端设计入手进行实验,因为自我评估问题在这里最为明显。在没有任何干预的情况下,Claude 通常会倾向于安全、可预测的布局——这些布局在技术上可用,但在视觉上平淡无奇。

I started by experimenting on frontend design, where the self-evaluation issue was most visible. Absent any intervention, Claude normally gravitates toward safe, predictable layouts that are technically functional but visually unremarkable.

有两点洞见塑造了我为前端设计构建的评测框架。第一,虽然美学无法完全归结为一个分数——而且个人品味总是因人而异——但可以通过编码了设计原则与偏好的评分标准来加以改进。“这个设计美吗?”很难得到一致的答案,但“这个设计是否遵循了我们关于优秀设计的原则?”则给了 Claude 一个具体可评分的依据。第二,通过将前端生成与前端评分分离,我们可以创建一个反馈回路,推动生成器产出更强的结果。

Two insights shaped the harness I built for frontend design. First, while aesthetics can't be fully reduced to a score—and individual tastes will always vary—they can be improved with grading criteria that encode design principles and preferences. "Is this design beautiful?" is hard to answer consistently, but "does this follow our principles for good design?" gives Claude something concrete to grade against. Second, by separating frontend generation from frontend grading, we can create a feedback loop that drives the generator toward stronger outputs.

基于这一思路,我编写了四条评分标准,并在提示词中同时提供给生成器智能体和评估器智能体:

With this in mind, I wrote four grading criteria that I gave to both the generator and evaluator agents in their prompts:

* **设计质量:** 设计是否感觉像一个连贯的整体,而非零散部件的堆砌?在这方面表现出色意味着色彩、字体排印、布局、图像及其他细节相互结合,营造出独特的情调与识别度。

* **Design quality:** Does the design feel like a coherent whole rather than a collection of parts? Strong work here means the colors, typography, layout, imagery, and other details combine to create a distinct mood and identity.

* **原创性:** 是否有自定义决策的痕迹,还是仅仅是模板布局、库默认值和 AI 生成的模式?人类设计师应当能识别出刻意的创意选择。未经修改的现成组件——或 AI 生成的明显迹象,比如白色卡片上的紫色渐变——在此项上不合格。

* **Originality:** Is there evidence of custom decisions, or is this template layouts, library defaults, and AI-generated patterns? A human designer should recognize deliberate creative choices. Unmodified stock components—or telltale signs of AI generation like purple gradients over white cards—fail here.

* **工艺:** 技术执行:排版层次、间距一致性、色彩和谐、对比度。这是能力检查而非创造力检查。大多数合理的实现在此默认表现良好;失败意味着基础不牢。

* **Craft:** Technical execution: typography hierarchy, spacing consistency, color harmony, contrast ratios. This is a competence check rather than a creativity check. Most reasonable implementations do fine here by default; failing means broken fundamentals.

* **功能性:** 独立于美学的可用性。用户能否理解界面的功能、找到主要操作并在不猜测的情况下完成任务?

* **Functionality:** Usability independent of aesthetics. Can users understand what the interface does, find primary actions, and complete tasks without guessing?

我强调设计质量和原创性,而非工艺和功能性。Claude 在工艺和功能性上默认已经得分很高,因为所需的技术能力往往对模型来说很自然。但在设计和原创性上,Claude 经常产生平庸的输出。评分标准明确惩罚高度通用的“AI 垃圾”模式,并通过更重地加权设计和原创性,推动模型进行更多美学冒险。

I emphasized design quality and originality over craft and functionality. Claude already scored well on craft and functionality by default, as the required technical competence tended to come naturally to the model. But on design and originality, Claude often produced outputs that were bland at best. The criteria explicitly penalized highly generic “AI slop” patterns, and by weighting design and originality more heavily it pushed the model toward more aesthetic risk-taking.

我使用带有详细分数分解的少样本示例来校准评估器。这确保了评估器的判断与我的偏好一致,并减少了迭代间的分数漂移。

I calibrated the evaluator using few-shot examples with detailed score breakdowns. This ensured the evaluator’s judgment aligned with my preferences, and reduced score drift across iterations.

我在 Claude Agent SDK 上构建了循环,这使得编排保持简单直接。生成器智能体首先根据用户提示创建 HTML/CSS/JS 前端。我给评估器提供了 Playwright MCP,使其能够在评分每个标准并撰写详细评论之前直接与实时页面交互。在实践中,评估器会自行导航页面,截图并仔细研究实现,然后产生评估。该反馈作为下一次迭代的输入流回生成器。每次生成我运行 5 到 15 次迭代,每次迭代通常推动生成器朝着更独特的方向发展,因为它回应评估器的批评。由于评估器是主动导航页面而不是对静态截图评分,每个周期都花费实际时间。完整运行长达四小时。我还指示生成器在每次评估后做出战略决策:如果分数趋势良好,则完善当前方向;如果方法不奏效,则转向完全不同的美学。

I built the loop on the Claude Agent SDK, which kept the orchestration straightforward. A generator agent first created an HTML/CSS/JS frontend based on a user prompt. I gave the evaluator the Playwright MCP, which let it interact with the live page directly before scoring each criterion and writing a detailed critique. In practice, the evaluator would navigate the page on its own, screenshotting and carefully studying the implementation before producing its assessment. That feedback flowed back to the generator as input for the next iteration. I ran 5 to 15 iterations per generation, with each iteration typically pushing the generator in a more distinctive direction as it responded to the evaluator's critique. Because the evaluator was actively navigating the page rather than scoring a static screenshot, each cycle took real wall-clock time. Full runs stretched up to four hours. I also instructed the generator to make a strategic decision after each evaluation: refine the current direction if scores were trending well, or pivot to an entirely different aesthetic if the approach wasn't working.

在多次运行中,评估者的评估随着迭代而改善,随后趋于平稳,但仍有余地。一些生成结果逐步改进,而另一些则在迭代之间出现了急剧的美学转变。

Across runs, the evaluator's assessments improved over iterations before plateauing, with headroom still remaining. Some generations refined incrementally. Others took sharp aesthetic turns between iterations.

标准的措辞以我未曾完全预料的方式引导了生成器。包含诸如“最佳设计应达到博物馆级品质”之类的短语,将设计推向特定的视觉趋同,这表明与标准相关的提示直接塑造了输出的特征。

The wording of the criteria steered the generator in ways I didn't fully anticipate. Including phrases like "the best designs are museum quality" pushed designs toward a particular visual convergence, suggesting that the prompting associated with the criteria directly shaped the character of the output.

虽然分数通常随着迭代而提高,但模式并不总是完全线性的。后期的实现总体上往往更好,但我经常看到一些情况,我更喜欢中间的迭代而不是最后一次。实现复杂度也往往随着轮次增加而增加,生成器根据评估者的反馈寻求更雄心勃勃的解决方案。即使在第一次迭代中,输出也明显优于完全没有提示的基线,这表明标准和相关语言本身引导模型远离了通用默认值,之后任何评估者的反馈才导致进一步的改进。

While scores generally improved over iterations, the pattern was not always cleanly linear. Later implementations tended to be better as a whole, but I regularly saw cases where I preferred a middle iteration over the last one. Implementation complexity also tended to increase across rounds, with the generator reaching for more ambitious solutions in response to the evaluator's feedback. Even on the first iteration, outputs were noticeably better than a baseline with no prompting at all, suggesting the criteria and associated language themselves steered the model away from generic defaults before any evaluator feedback led to further refinement.

在一个显著的例子中,我提示模型为一个荷兰艺术博物馆创建一个网站。到第九次迭代时,它为一个虚构的博物馆生成了一个干净、深色主题的着陆页。该页面视觉上很精致,但很大程度上符合我的预期。然后,在第十个周期,它完全放弃了这个方法,将网站重新构想为一种空间体验:一个用 CSS 透视渲染的棋盘格地板的 3D 房间,艺术品以自由形式的位置挂在墙上,并且通过门道在画廊房间之间导航,而不是滚动或点击。这是我在单次生成中从未见过的创造性飞跃。

In one notable example, I prompted the model to create a website for a Dutch art museum. By the ninth iteration, it had produced a clean, dark-themed landing page for a fictional museum. The page was visually polished but largely in line with my expectations. Then, on the tenth cycle, it scrapped the approach entirely and reimagined the site as a spatial experience: a 3D room with a checkered floor rendered in CSS perspective, artwork hung on the walls in free-form positions, and doorway-based navigation between gallery rooms instead of scroll or click. It was the kind of creative leap that I hadn't seen before from a single-pass generation.

扩展至全栈编程 Scaling to full-stack coding

基于这些发现,我将这种受 GAN 启发的模式应用于全栈开发。生成器-评估器循环自然地映射到软件开发生命周期中:在这一周期里,代码审查与 QA 所起的结构性作用,与设计评估器完全相同。

With these findings in hand, I applied this GAN-inspired pattern to full-stack development. The generator-evaluator loop maps naturally onto the software development lifecycle, where code review and QA serve the same structural role as the design evaluator.

架构 The architecture

在我们早期长时间运行的 harness 中,我们通过一个初始化智能体、一个每次只处理一个功能的编码智能体,以及会话之间的上下文重置,解决了连贯的多会话编码问题。上下文重置是一个关键的突破口:该 harness 使用的是 Sonnet 4.5,它表现出了前文提到的“上下文焦虑”倾向。构建一个能在上下文重置之间良好工作的 harness,是让模型保持任务专注的关键。Opus 4.5 在很大程度上自行消除了这一行为,因此我能够从这个 harness 中完全去掉上下文重置。在整个构建过程中,各智能体作为一次连续的会话运行,由 Claude Agent SDK 的自动压缩机制处理沿途的上下文增长。

In our earlier long-running harness, we had solved for coherent multi-session coding with an initializer agent, a coding agent that worked one feature at a time, and context resets between sessions. Context resets were a key unlock: the harness used Sonnet 4.5, which exhibited the "context anxiety" tendency mentioned earlier. Creating a harness that worked well across context resets was key to keeping the model on task. Opus 4.5 largely removed that behavior on its own, so I was able to drop context resets from this harness entirely. The agents were run as one continuous session across the whole build, with the Claude Agent SDK's automatic compaction handling context growth along the way.

在这项工作中,我在原始 harness 的基础上构建了一个三智能体系统,每个智能体针对我在先前运行中观察到的一个特定缺口。该系统包含以下智能体角色:

For this work I built on the foundation from the original harness with a three-agent system, with each agent addressing a specific gap I'd observed in prior runs. The system contained the following agent personas:

**规划者(Planner):** 我们之前长时间运行的 harness 要求用户预先提供详细的规格说明。我希望将这一步自动化,因此创建了一个规划者智能体,它接收一个简单的 1-4 句提示,并将其扩展为完整的产品规格。我提示它在范围上要有雄心,并专注于产品上下文和高层技术设计,而非详细的技术实现。这一强调源于一种担忧:如果规划者试图预先指定细粒度的技术细节并出了错,规格中的错误会级联到下游实现中。更明智的做法似乎是对智能体要产出的交付物加以约束,让它们在工作过程中自行摸索路径。我还要求规划者寻找机会将 AI 功能融入产品规格中。(参见文末附录中的示例。)

**Planner:** Our previous long-running harness required the user to provide a detailed spec upfront. I wanted to automate that step, so I created a planner agent that took a simple 1-4 sentence prompt and expanded it into a full product spec. I prompted it to be ambitious about scope and to stay focused on product context and high-level technical design rather than detailed technical implementation. This emphasis was due to the concern that if the planner tried to specify granular technical details upfront and got something wrong, the errors in the spec would cascade into the downstream implementation. It seemed smarter to constrain the agents on the deliverables to be produced and let them figure out the path as they worked. I also asked the planner to find opportunities to weave AI features into the product specs. (See example in the Appendix at the bottom.)

**生成者(Generator):** 早期 harness 中一次只处理一个功能的方法在范围管理上效果良好。我在这里应用了类似的模式,指示生成者以冲刺(sprint)方式工作,每次从规格中选取一个功能。每个冲刺使用 React、Vite、FastAPI 和 SQLite(后来是 PostgreSQL)技术栈实现应用,并指示生成者在每个冲刺结束时先自我评估其工作,然后再移交给 QA。它还有 git 用于版本控制。

**Generator:** The one-feature-at-a-time approach from the earlier harness worked well for scope management. I applied a similar model here, instructing the generator to work in sprints, picking up one feature at a time from the spec. Each sprint implemented the app with a React, Vite, FastAPI, and SQLite (later PostgreSQL) stack, and the generator was instructed to self-evaluate its work at the end of each sprint before handing off to QA. It also had git for version control.

**评估者(Evaluator):** 早期 harness 生成的应用往往看起来令人印象深刻,但当你真正尝试使用时仍然存在真实缺陷。为了捕捉这些缺陷,评估者使用 Playwright MCP 以用户的方式点击浏览正在运行的应用,测试 UI 功能、API 端点和数据库状态。然后,它根据发现的缺陷以及一套仿照前端实验建模的标准对每个冲刺进行评分,这些标准在此经过调整以涵盖产品深度、功能、视觉设计和代码质量。每个标准都有一个硬性阈值,如果任何一项低于阈值,该冲刺即告失败,生成者会收到关于问题所在的详细反馈。

**Evaluator:** Applications from earlier harnesses often looked impressive but still had real bugs when you actually tried to use them. To catch these, the evaluator used the Playwright MCP to click through the running application the way a user would, testing UI features, API endpoints, and database states. It then graded each sprint against both the bugs it had found and a set of criteria modeled on the frontend experiment, adapted here to cover product depth, functionality, visual design, and code quality. Each criterion had a hard threshold, and if any one fell below it, the sprint failed and the generator got detailed feedback on what went wrong.

在每个 sprint 开始之前,生成器与评估器会协商一份 sprint 契约:在编写任何代码之前,就先就该块工作的“完成”标准达成一致。之所以设置这一环节,是因为产品规格书有意保持在高层次,我希望有一个步骤来弥合用户故事与可测试实现之间的差距。生成器提出它将构建什么以及如何验证成功,评估器则审查该提案,以确保生成器构建的是正确的东西。双方反复迭代,直至达成一致。

Before each sprint, the generator and evaluator negotiated a sprint contract: agreeing on what "done" looked like for that chunk of work before any code was written. This existed because the product spec was intentionally high-level, and I wanted a step to bridge the gap between user stories and testable implementation. The generator proposed what it would build and how success would be verified, and the evaluator reviewed that proposal to make sure the generator was building the right thing. The two iterated until they agreed.

沟通通过文件进行:一个智能体写入文件,另一个智能体读取该文件,并在该文件内作出回应,或新建一个文件供前一个智能体依次读取。随后,生成器依据商定的契约进行构建,再将工作移交给 QA。这样既能让工作忠实于规格书,又不会过早地对实现细节规定得过于具体。

Communication was handled via files: one agent would write a file, another agent would read it and respond either within that file or with a new file that the previous agent would read in turn. The generator then built against the agreed-upon contract before handing the work off to QA. This kept the work faithful to the spec without over-specifying implementation too early.

运行测试框架 Running the harness

对于这个测试框架的第一个版本,我使用了 Claude Opus 4.5,将用户提示分别输入完整测试框架和单智能体系统以进行对比。我之所以使用 Opus 4.5,是因为在我开始这些实验时,它是我们最好的编程模型。

For the first version of this harness, I used Claude Opus 4.5, running user prompts against both the full harness and a single-agent system for comparison. I used Opus 4.5 since it was our best coding model when I began these experiments.

我编写了以下提示词来生成一个复古电子游戏制作器:

I wrote the following prompt to generate a retro video game maker:

_创建一个 2D 复古游戏制作器,功能包括关卡编辑器、精灵编辑器、实体行为和可玩的测试模式。_

_Create a 2D retro game maker with features including a level editor, sprite editor, entity behaviors, and a playable test mode._

下表展示了测试框架类型、运行时长和总成本。

The table below shows the harness type, the length of time it ran, and the total cost.

该测试框架的成本高出 20 倍以上,但输出质量的差异立竿见影。

The harness was over 20x more expensive, but the difference in output quality was immediately apparent.

我原本期望有一个界面,可以在其中构建关卡及其组成部分(精灵、实体、瓦片布局),然后点击播放来实际游玩该关卡。我先打开了 solo run 的输出,初始应用看起来符合这些预期。

I was expecting an interface where I could construct a level and its component parts (sprites, entities, tile layout) and then hit play to actually play the level. I started by opening the solo run's output, and the initial application seemed in line with those expectations.

然而,当我逐项点击时,问题开始浮现。布局浪费空间,固定高度的面板使大部分视口空置。工作流僵化。尝试填充关卡时,系统提示我先创建精灵和实体,但 UI 中没有任何引导指向这一顺序。更关键的是,实际游戏是坏的。我的实体出现在屏幕上,但没有任何东西响应输入。深入代码后发现,实体定义与游戏运行时之间的连接是断的,而界面上没有任何迹象表明问题出在哪里。

As I clicked through, however, issues started to emerge. The layout wasted space, with fixed-height panels leaving most of the viewport empty. The workflow was rigid. Trying to populate a level prompted me to create sprites and entities first, but nothing in the UI guided me toward that sequence. More to the point, the actual game was broken. My entities appeared on screen but nothing responded to input. Digging into the code revealed that the wiring between entity definitions and the game runtime was broken, with no surface indication of where.

在评估了单独运行(solo run)之后,我将注意力转向了 harness 运行。这次运行同样从一句提示词开始,但规划器(planner)步骤将该提示词扩展为一份包含 16 项功能、横跨十个冲刺(sprint)的规格说明。它远远超出了单独运行所尝试的范围。除了核心编辑器和游玩模式外,规格还要求一个精灵动画系统、行为模板、音效与音乐、一个 AI 辅助的精灵生成器和关卡设计器,以及带可分享链接的游戏导出功能。我让规划器访问了我们的前端设计技能,它阅读并利用该技能为应用创建了一套视觉设计语言,并将其作为规格的一部分。对于每个冲刺,生成器(generator)和评估器(evaluator)协商一份契约,定义该冲刺的具体实现细节,以及用于验证完成情况的可测试行为。

After evaluating the solo run, I turned my attention to the harness run. This run started from the same one-sentence prompt, but the planner step expanded that prompt into a 16-feature spec spread across ten sprints. It went well beyond what the solo run attempted. In addition to the core editors and play mode, the spec called for a sprite animation system, behavior templates, sound effects and music, an AI-assisted sprite generator and level designer, and game export with shareable links. I gave the planner access to our frontend design skill, which it read and used to create a visual design language for the app as part of the spec. For each sprint, the generator and evaluator negotiated a contract defining the specific implementation details for the sprint, and the testable behaviors that would be tested to verify completion.

该应用立刻表现出比单独运行更高的完成度和流畅度。画布使用了整个视口,面板尺寸合理,界面具有一致的视觉识别,与规格中的设计方向保持一致。单独运行中见到的一些笨拙之处确实仍然存在——工作流仍然没有明确说明应该先构建精灵和实体,再尝试填充关卡,我不得不通过摸索来弄明白这一点。这更像是基础模型产品直觉上的缺口,而非 harness 被设计来解决的问题,不过它确实提示了在 harness 内部进行有针对性的迭代可能有助于进一步提升输出质量。

The app immediately showed more polish and smoothness than the solo run. The canvas used the full viewport, the panels were sized sensibly, and the interface had a consistent visual identity that tracked the design direction from the spec. Some of the clunkiness I'd seen in the solo run did remain—the workflow still didn't make it clear that you should build sprites and entities before trying to populate a level, and I had to figure that out by poking around. This read as a gap in the base model's product intuition rather than something the harness was designed to address, though it did suggest a place where targeted iteration inside the harness could help to further improve output quality.

在使用各个编辑器时,新运行相对于单独运行的优势变得更加明显。精灵编辑器更丰富、功能更完整,工具面板更整洁,颜色选择器更好,缩放控制也更易用。

Working through the editors, the new run's advantages over solo became more apparent. The sprite editor was richer and more fully featured, with cleaner tool palettes, a better color picker, and more usable zoom controls.

因为我曾要求规划器将 AI 功能融入其规格,所以该应用还内置了 Claude 集成,让我可以通过提示词生成游戏的不同部分。这显著加快了工作流。

Because I'd asked the planner to weave AI features into its specs, the app also came with a built-in Claude integration that let me generate different parts of the game through prompting. This significantly sped up the workflow.

开始界面 精灵编辑器 AI 游戏设计 AI 游戏设计 游戏游玩

Opening screen Sprite editor AI game design AI game design Game play

使用内置 AI 功能生成关卡

Using the built-in AI feature to generate the level

最大的差异在于游玩模式。我实际上能够移动我的实体并玩游戏。物理效果有一些粗糙之处——我的角色跳上了一个平台,但最终与平台重叠,这感觉上不对——但核心功能是有效的,而单独运行未能做到这一点。在移动了一会儿后,我确实遇到了 AI 游戏关卡构建的一些限制。有一堵大墙我无法跳过,所以我卡住了。这表明测试框架可以处理一些常识性改进和边缘情况,以进一步完善应用。

The biggest difference was in play mode. I was actually able to move my entity and play the game. The physics had some rough edges—my character jumped onto a platform but ended up overlapping with it, which felt intuitively wrong—but the core thing worked, which the solo run did not manage. After moving around a bit, I did hit some limitations with the AI’s game level construction. There was a large wall that I wasn’t able to jump past, so I was stuck. This suggested there were some common sense improvements and edge cases that the harness could handle to further refine the app.

通读日志后可以清楚地看到,评估器始终使实现与规范保持一致。每个冲刺,它都会遍历冲刺合同的测试标准,并通过 Playwright 运行应用程序,对任何偏离预期行为的地方提交缺陷报告。合同非常细粒度——仅 Sprint 3 就有 27 条覆盖关卡编辑器的标准——而且评估器的发现足够具体,无需额外调查即可采取行动。下表展示了我们的评估器识别出的几个问题示例:

Reading through the logs, it was clear that the evaluator kept the implementation in line with the spec. Each sprint, it walked through the sprint contract's test criteria and exercised the running application through Playwright, filing bugs against anything that diverged from expected behavior. The contracts were granular—Sprint 3 alone had 27 criteria covering the level editor—and the evaluator's findings were specific enough to act on without extra investigation. The table below shows several examples of issues our evaluator identified:

矩形填充工具允许点击拖动以用选中的图块填充矩形区域

Rectangle fill tool allows click-drag to fill a rectangular area with selected tile

**失败** —该工具仅在拖动起点/终点放置图块,而不是填充整个区域。\(\text{fillRectangle}\) 函数存在,但在 \(\text{mouseUp}\) 时未正确触发。

**FAIL** — Tool only places tiles at drag start/end points instead of filling the region. \(\text{fillRectangle}\) function exists but isn't triggered properly on \(\text{mouseUp}\).

用户可以选择并删除已放置的实体生成点

User can select and delete placed entity spawn points

**失败** — \(\text{LevelEditor.tsx:892}\) 处的删除键处理程序要求同时设置 \(\text{selection}\) 和 \(\text{selectedEntityId}\),但点击实体仅设置 \(\text{selectedEntityId}\)。条件应为 \(\text{selection} \|\| (\text{selectedEntityId} \&\& \text{activeLayer} === \text{'entity'})\)。

**FAIL** — Delete key handler at \(\text{LevelEditor.tsx:892}\) requires both \(\text{selection}\) and \(\text{selectedEntityId}\) to be set, but clicking an entity only sets \(\text{selectedEntityId}\). Condition should be \(\text{selection} \|\| (\text{selectedEntityId} \&\& \text{activeLayer} === \text{'entity'})\).

用户可以通过 API 重新排序动画帧

User can reorder animation frames via API

**失败** —PUT /frames/reorder 路由定义在 /{frame_id} 路由之后。FastAPI 将 'reorder' 匹配为 frame_id 整数并返回 422:“unable to parse string as an integer.”

**FAIL** — The PUT /frames/reorder route is defined after the /{frame_id} routes. FastAPI matches 'reorder' as a frame_id integer and returns 422: "unable to parse string as an integer."

让评估器达到这一水平并非易事。开箱即用时,Claude 是一个糟糕的 QA 智能体。在早期运行中,我看着它识别出合理的问题,然后说服自己这些问题无关紧要,最终还是批准了工作。它还倾向于进行表面测试,而不是探查边缘情况,因此更微妙的 bug 常常被漏掉。调优循环是阅读评估器的日志,找到其判断与我的判断出现分歧的例子,并更新 QA 的提示词以解决这些问题。经过几轮这样的开发循环,评估器的评分方式才让我觉得合理。即便如此,测试框架的输出仍显示出模型 QA 能力的局限:小的布局问题、某些地方交互感觉不直观,以及评估器未彻底检验的更深层嵌套功能中未被发现的 bug。显然,通过进一步调优还有更多验证空间可以挖掘。但与单独运行相比——在单独运行中,应用程序的核心功能根本无法工作——提升是显而易见的。

Getting the evaluator to perform at this level took work. Out of the box, Claude is a poor QA agent. In early runs, I watched it identify legitimate issues, then talk itself into deciding they weren't a big deal and approve the work anyway. It also tended to test superficially, rather than probing edge cases, so more subtle bugs often slipped through. The tuning loop was to read the evaluator's logs, find examples where its judgment diverged from mine, and update the QA's prompt to solve for those issues. It took several rounds of this development loop before the evaluator was grading in a way that I found reasonable. Even then, the harness output showed the limits of the model's QAing capabilities: small layout issues, interactions that felt unintuitive in places, and undiscovered bugs in more deeply nested features that the evaluator hadn't exercised thoroughly. There was clearly more verification headroom to capture with further tuning. But compared to the solo run, where the central feature of the application simply didn't work, the lift was obvious.

对 harness 的迭代 Iterating on the harness

第一组 harness 结果令人鼓舞,但它同时也笨重、缓慢且昂贵。合乎逻辑的下一步是找到简化 harness 而不降低其性能的方法。这在一定程度上是常识,在一定程度上也源于一个更普遍的原则:harness 中的每个组件都编码了一个关于模型自身无法完成什么的假设,而这些假设值得进行压力测试——既因为它们可能是错误的,也因为随着模型改进,它们可能很快过时。我们的博客文章《Building Effective Agents》将这一基本思想表述为“找到尽可能简单的解决方案,只在需要时才增加复杂性”,这也是任何维护智能体 harness 的人都会反复遇到的模式。

The first set of harness results was encouraging, but it was also bulky, slow, and expensive. The logical next step was to find ways to simplify the harness without degrading its performance. This was partly common sense and partly a function of a more general principle: every component in a harness encodes an assumption about what the model can't do on its own, and those assumptions are worth stress testing, both because they may be incorrect, and because they can quickly go stale as models improve. Our blog post Building Effective Agents frames the underlying idea as "find the simplest solution possible, and only increase complexity when needed," and it's a pattern that shows up consistently for anyone maintaining an agent harness.

在我第一次尝试简化时,我大幅削减了 harness,并尝试了一些有创意的新想法,但未能复现原始版本的性能。同时也变得难以判断 harness 设计中哪些部分真正起支撑作用,以及以何种方式起作用。基于那次经验,我转向一种更有条理的方法:一次移除一个组件,并审视它对最终结果产生了什么影响。

In my first attempt to simplify, I cut the harness back radically and tried a few creative new ideas, but I wasn't able to replicate the performance of the original. It also became difficult to tell which pieces of the harness design were actually load-bearing, and in what ways. Based on that experience, I moved to a more methodical approach, removing one component at a time and reviewing what impact it had on the final result.

在我进行这些迭代循环的同时,我们还发布了 Opus 4.6,这为降低 harness 复杂性提供了进一步动力。有充分理由预期 4.6 所需的脚手架(scaffolding)会比 4.5 更少。摘自我们的发布博客:“[Opus 4.6] 规划更谨慎,能更长时间地持续执行智能体式任务,在更大的代码库中运行更可靠,并具备更好的代码审查和调试能力以发现自身错误。”它在长上下文检索方面也有大幅提升。这些都是 harness 原本被构建来补充的能力。

As I was going through these iteration cycles, we also released Opus 4.6, which provided further motivation to reduce harness complexity. There was good reason to expect 4.6 would need less scaffolding than 4.5 did. From our launch blog: "[Opus 4.6] plans more carefully, sustains agentic tasks for longer, can operate more reliably in larger codebases, and has better code review and debugging skills to catch its own mistakes." It also improved substantially on long-context retrieval. These were all capabilities the harness had been built to supplement.

移除冲刺构造 Removing the sprint construct

我首先完全移除了冲刺构造。冲刺结构曾有助于将工作分解成块,以便模型连贯地工作。鉴于 Opus 4.6 的改进,有充分理由相信模型可以原生处理这项工作,而无需这种分解。

I started by removing the sprint construct entirely. The sprint structure had helped to decompose work into chunks for the model to work coherently. Given the improvements in Opus 4.6, there was good reason to believe that the model could natively handle the job without this sort of decomposition.

我保留了规划器和评估器,因为两者都持续带来明显价值。没有规划器,生成器会范围不足:给定原始提示,它会不先规划工作就开始构建,最终创建出比规划器功能更少的应用。

I kept both the planner and evaluator, as each continued to add obvious value. Without the planner, the generator under-scoped: given the raw prompt, it would start building without first speccing its work, and end up creating a less feature-rich application than the planner did.

移除冲刺构造后,我将评估器改为在运行结束时单次执行,而不是每个冲刺评分。由于模型能力更强,这改变了评估器在某些运行中的承重程度,其有用性取决于任务相对于模型能可靠独立完成的范围。在 4.5 上,那个边界很近:我们的构建处于生成器单独能做好范围的边缘,评估器在整个构建过程中捕捉到了有意义的问题。在 4.6 上,模型的原始能力增加,因此边界向外移动。过去需要评估器检查才能连贯实现的任务,现在通常处于生成器单独处理良好的范围内,对于该边界内的任务,评估器成为不必要的开销。但对于构建中仍处于生成器能力边缘的部分,评估器继续提供真正的提升。

With the sprint construct removed, I moved the evaluator to a single pass at the end of the run rather than grading per sprint. Since the model was much more capable, it changed how load-bearing the evaluator was for certain runs, with its usefulness depending on where the task sat relative to what the model could do reliably on its own. On 4.5, that boundary was close: our builds were at the edge of what the generator could do well solo, and the evaluator caught meaningful issues across the build. On 4.6, the model's raw capability increased, so the boundary moved outward. Tasks that used to need the evaluator's check to be implemented coherently were now often within what the generator handled well on its own, and for tasks within that boundary, the evaluator became unnecessary overhead. But for the parts of the build that were still at the edge of the generator’s capabilities, the evaluator continued to give real lift.

实际意义是,评估器不是固定的是或否决策。当任务超出当前模型可靠独立完成的范围时,它值得付出成本。

The practical implication is that the evaluator is not a fixed yes-or-no decision. It is worth the cost when the task sits beyond what the current model does reliably solo.

除了结构简化,我还添加了提示来改进测试框架将 AI 功能构建到每个应用中的方式,特别是让生成器构建一个能通过工具驱动应用自身功能的合适智能体。这需要真正的迭代,因为相关知识足够新,Claude 的训练数据覆盖很薄。但经过足够调优,生成器正确构建了智能体。

Alongside the structural simplification, I also added prompting to improve how the harness built AI features into each app, specifically getting the generator to build a proper agent that could drive the app's own functionality through tools. That took real iteration, since the relevant knowledge is recent enough that Claude's training data covers it thinly. But with enough tuning, the generator was building agents correctly.

更新后测试框架的结果 Results from the updated harness

为了检验更新后的测试框架,我使用以下提示词来生成一个数字音频工作站(DAW),即一款用于作曲、录音和混音的音乐制作程序:

To put the updated harness to the test, I used the following prompt to generate a Digital Audio Workstation (DAW), a music production program for composing, recording, and mixing songs:

_使用 Web Audio API 在浏览器中构建一个功能齐全的 DAW。_

_Build a fully featured DAW in the browser using the Web Audio API._

这次运行仍然耗时且昂贵,大约 4 小时,token 成本为 124 美元。

The run was still lengthy and expensive, at about 4 hours and $124 in token costs.

大部分时间花在构建器上,它连贯运行了两个多小时,且无需 Opus 4.5 曾需要的冲刺式分解。

Most of the time went to the builder, which ran coherently for over two hours without the sprint decomposition that Opus 4.5 had needed.

与之前的测试框架一样,规划器将一行提示词扩展为完整规格。从日志中可以看出,生成器模型在规划应用和智能体设计、连接智能体以及在移交给 QA 之前进行测试方面都做得很好。

As with the previous harness, the planner expanded the one-line prompt into a full spec. From the logs, I could see the generator model did a good job planning the app and the agent design, wiring the agent up, and testing it before handing off to QA.

尽管如此,QA 智能体仍然发现了真实的差距。在其第一轮反馈中,它指出:

That being said, the QA agent still caught real gaps. In its first-round feedback, it noted:

这是一个强大的应用,具有出色的设计保真度、可靠的 AI 智能体和良好的后端。主要失败点是功能完整性——虽然应用看起来令人印象深刻,AI 集成效果良好,但几个核心 DAW 功能仅用于显示,没有交互深度:剪辑无法在时间轴上拖动/移动,没有乐器 UI 面板(合成器旋钮、鼓垫),也没有视觉效果编辑器(EQ 曲线、压缩器表)。这些不是边缘情况——它们是使 DAW 可用的核心交互,而规范明确要求它们。

This is a strong app with excellent design fidelity, solid AI agent, and good backend. The main failure point is Feature Completeness — while the app looks impressive and the AI integration works well, several core DAW features are display-only without interactive depth: clips can't be dragged/moved on the timeline, there are no instrument UI panels (synth knobs, drum pads), and no visual effect editors (EQ curves, compressor meters). These aren't edge cases — they're the core interactions that make a DAW usable, and the spec explicitly calls for them.

在其第二轮反馈中,它再次发现了几个功能差距:

In its second round feedback, it again caught several functionality gaps:

剩余差距:

Remaining gaps:

- 音频录制仍然只是存根(按钮切换但没有麦克风捕获)

- Audio recording is still stub-only (button toggles but no mic capture)

- 未实现通过边缘拖拽调整片段大小以及片段分割功能

- Clip resize by edge drag and clip split not implemented

- 效果可视化采用数值滑块而非图形化呈现(无 EQ 曲线)

- Effect visualizations are numeric sliders, not graphical (no EQ curve)

当放任生成器自行其是时,它仍容易遗漏细节或只实现功能桩,而 QA 在捕捉这些最后一公里问题以供生成器修复方面仍然具有价值。

The generator was still liable to miss details or stub features when left to its own devices, and the QA still added value in catching those last-mile issues for the generator to fix.

根据提示词,我原本期望得到一个程序:在其中可以创作旋律、和声与鼓点模式,将它们编排成一首歌曲,并在过程中获得集成智能体的帮助。下方视频展示了结果。

Based on the prompt, I was expecting a program where I could create melodies, harmonies, and drum patterns, arrange them into a song, and get help from an integrated agent along the way. The video below shows the result.

该应用远非专业的音乐制作程序,智能体的歌曲创作能力显然还有很大提升空间。此外,Claude 实际上无法听到声音,这使得 QA 反馈回路在音乐品味方面的效果大打折扣。

The app is far from a professional music production program, and the agent's song composition skills could clearly use a lot of work. Additionally, Claude can't actually hear, which made the QA feedback loop less effective with respect to musical taste.

但这个最终应用已经具备了一个可用音乐制作程序的全部核心组件:在浏览器中运行的编曲视图、混音器和走带控制。除此之外,我完全通过提示词就拼出了一段简短的歌曲片段:智能体设定了速度和调性,铺了一段旋律,构建了鼓轨,调整了混音器电平,并添加了混响。歌曲创作的核心原语已经具备,智能体能够自主驱动它们,使用工具端到端地完成一个简单的制作。你或许会说它还不够完美——但它正在渐入佳境。

But the final app had all the core pieces of a functional music production program: a working arrangement view, mixer, and transport running in the browser. Beyond that, I was able to put together a short song snippet entirely through prompting: the agent set the tempo and key, laid down a melody, built a drum track, adjusted mixer levels, and added reverb. The core primitives for song composition were present, and the agent could drive them autonomously, using tools to create a simple production from end to end. You might say it's not pitch-perfect yet—but it's getting there.

接下来会怎样 What comes next

随着模型不断改进,我们可以大致预期它们能够工作更长时间、处理更复杂的任务。在某些情况下,这意味着围绕模型的脚手架随时间推移变得不那么重要,开发者可以等待下一个模型,看着某些问题自行解决。另一方面,模型越好,开发能够完成超出模型基线能力的复杂任务的 harness 的空间就越大。

As models continue to improve, we can roughly expect them to be capable of working for longer, and on more complex tasks. In some cases, that will mean the scaffold surrounding the model matters less over time, and developers can wait for the next model and see certain problems solve themselves. On the other hand, the better the models get, the more space there is to develop harnesses that can achieve complex tasks beyond what the model can do at baseline.

考虑到这一点,这项工作中有几点经验值得延续。始终良好的做法是:针对你正在构建的模型进行实验,阅读它在真实问题上的轨迹,并调整其性能以实现你期望的结果。在处理更复杂的任务时,有时通过分解任务并对问题的每个方面应用专门的智能体,可以获得提升空间。而当新模型发布时,通常良好的做法是重新审视 harness,剥离那些不再对性能起关键支撑作用的部分,并添加新的部分以实现以前可能无法达到的更强能力。

With this in mind, there are a few lessons from this work worth carrying forward. It is always good practice to experiment with the model you're building against, read its traces on realistic problems, and tune its performance to achieve your desired outcomes. When working on more complex tasks, there is sometimes headroom from decomposing the task and applying specialized agents to each aspect of the problem. And when a new model lands, it is generally good practice to re-examine a harness, stripping away pieces that are no longer load-bearing to performance and adding new pieces to achieve greater capability that may not have been possible before.

从这项工作中,我坚信:有趣的 harness 组合空间并不会随着模型改进而缩小。相反,它会转移,而 AI 工程师的有趣工作就是不断寻找下一个新颖的组合。

From this work, my conviction is that the space of interesting harness combinations doesn't shrink as models improve. Instead, it moves, and the interesting work for AI engineers is to keep finding the next novel combination.

互动版:针对本篇提问 →

接着读 · Read next