Artificial Analysis v5 and the benchmarker's dilemma
打开互动全文版(逐段中英对照 + 针对本篇提问)→本文考察了 Artificial Analysis 的 Intelligence Index v5——前沿语言模型最知名的独立基准——以及基准创建者所面临的更广泛困境。文章认为,尽管将模型质量简化为单一数字对于讨论和比较是必要的,但每一种聚合方法都涉及有意义的权衡,而基准设计不可避免地反映了一种可被调整以支持某种叙事的世界观。作者分析了 AAv5 在知识工作、编程、科学和通用能力上的权重,指出其面向前沿的立场扩大了闭源与开放权重模型之间的明显差距,尽管预计开放实验室将在未来的模型周期中缩小差距。文章判断,向私有基准的转变在方向上是合理的,但代价是失去社区审视。结论是,没有单一指数能够完美,该领域需要更多带有不同偏见的指数,而对基准测试的信任最终取决于透明度以及每位创建者所接受的权衡。
This article examines Artificial Analysis's Intelligence Index v5, the most prominent independent benchmark for frontier language models, and the broader dilemma facing benchmark creators. It argues that while reducing model quality to a single number is necessary for discussion and comparison, every aggregation method involves meaningful trade-offs, and benchmark design inevitably reflects a worldview that can be tuned to support a narrative. The author analyzes AAv5's weighting across knowledge work, coding, science, and general capabilities, noting that its frontier-oriented stance widens the apparent gap between closed and open-weight models, though open labs are expected to close ground in coming model cycles. The shift toward private benchmarks is judged directionally defensible but costly in lost community scrutiny. The conclusion is that no single index can be perfect, and the field needs more indices with differing biases, while trust in benchmarking ultimately rests on transparency and the trade-offs each creator accepts.
爬山容易,知道该爬什么山却很难。让分数上升是**昂贵的**,但相比在当今 AI 的真正前沿创建有意义的基准测试,我们对此有更清晰的行动手册。
Hill-climbing is easy, knowing what to hill-climb is hard. Making scores go up is _expensive_, but we have a much clearer playbook for it than we do for creating meaningful benchmarks at the true frontier of AI today.
语言模型的独立评估任务之艰巨被低估了。这些团队并非只是拿当前模型来认证其分数——那本身也是一项技术性很强的任务。如果 AI 进展缓慢,这就是他们的工作。当 AI 进展如此之快时,这项工作的技术细节就成了试图用一个数字来定义当今优秀 AI 模型的常规部分。目标是得到一个能代表当前模型的单一数字,与直觉检查、社区情绪和科学分析相匹配,同时具有足够的前瞻性,以免在 3 个月内过时。
Independent evaluation of language models is under-appreciated in how hard of a task it is. These groups are not just taking current models and certifying their scores, which is a technically involved task. This is their job if AI progress is slow. When AI progress is as fast as it is, the technical details of the job are the routine work part of trying to put a single number on what makes for a good AI model today. The goal is to have a single number that is representative of current models, matching vibe checks, community sentiment, and scientific analysis, while being forward looking enough so that it’s not outdated in 3 months.
每个构建基准测试的人都面临一种悖论。你需要你的基准测试归结为一个单一数字,以便于讨论、用于报告和比较模型,但每种聚合分数的方法都伴随着有意义的权衡。最优秀的科学家需要这样做,每个人都需要。关键在于创建你认为最公平的聚合指标,同时知道总会有不完美之处。
Everyone building a benchmark faces a sort of paradox. You need your benchmark to reduce to one single number, so that it is easy to discuss, use in reports, and compare models, but every method of aggregating scores comes with meaningful trade-offs. The best scientists need to do this, everyone does. It’s a matter of creating aggregation metrics that you think are most fair, knowing there will always be imperfections.
鉴于当今 AI 生态系统中的资金规模,独立评估的权重更高——你决定着一个万亿美元市场的赢家和输家。你还影响一些模型本身,引导进步的方向。这压力很大!
With the amount of money in the AI ecosystem today, the weight of independent evaluation is even higher – you dictate the winners and losers of a trillion dollar market. You also influence some of the models themselves, steering the shape of progress. This is a lot of pressure!
我写这篇文章的由头是 Artificial Analysis 最近更新了他们的 Intelligence Index v5。AA 指数是当前前沿模型最突出的独立智能指标,紧随其后的是 Epoch AI 的能力指数(ECI),但差距很大。迄今为止,所有这些指标对社区都极其有用,同时也存在巨大缺陷!
My prompt for this article is Artificial Analysis’s recent update to their Intelligence Index, v5. The AA index is the most prominent independent intelligence metric for the latest frontier models, followed in a distant second by Epoch AI’s Capabilities Index (ECI). To date, all of these have been incredibly useful for the community and massively flawed!
Interconnects AI 是一份由读者支持的出版物。欢迎考虑成为订阅者。
Interconnects AI is a reader-supported publication. Consider becoming a subscriber.
如今的语言模型已具备如此多的能力,且能力范围正在快速扩大。我们这种单一智能指数的时代 _理应_ 很快被淘汰,但这将需要很长时间。
Language models are capable of so much today, and this scope of capabilities is growing quickly. Our era of singular intelligence indices _should_ be phased out soon, but it will take a long time.
以下是来自其博客文章的 AAv5 摘要,文中将其描述为“反映当今真实世界的使用情况:”
A summary of AAv5 is below, from their blog post, where they describe it as "reflecting real-world usage today:"
**Intelligence Index v5 权重:**
**Intelligence Index v5 weights:**
**➤ 知识工作(25%):** AA-Briefcase(15%)、AutomationBench-AA(5%)和 GDP.pdf(5%)
**➤ Knowledge Work (25%):** AA-Briefcase (15%), AutomationBench-AA (5%), and GDP.pdf (5%)
**➤ 编程(25%):** FrontierCode 2.0(15%)与 Terminal-Bench 4.0(10%)
**➤ Coding (25%):** FrontierCode 2.0 (15%) and Terminal-Bench 4.0 (10%)
**➤ 科学(25%):** Terminal-Bench-Science(15%)与 CritPt v2 困难子集(10%)
**➤ Science (25%):** Terminal-Bench-Science (15%) and CritPt v2 hard subset (10%)
**➤ 通用(25%):** AA-Omniscience v1.1 占 15%(10% 准确率、5% 非幻觉率)与 HLE-Diamond(10%)
**➤ General (25%):** AA-Omniscience v1.1 15% (10% accuracy, 5% non-hallucination) and HLE-Diamond (10%)
乍看之下,从总体上看,他们的模型排名确实显得合理。不过,正如任何基准测试都会出现的那样,它也显露出吃力和混乱的迹象——GPT 6.1 Terra 在 AAv5 上击败了 Claude Fable 5.1。如今,一些基准测试衡量的是能力的绝对前沿,通过你偏好的 harness 在开放式任务中表现良好,而这通常确实需要一个大模型。另一些基准测试衡量的则是模型在重复性智能体式任务中所需要的某些技能。我感觉 AA 正在把他们的基准测试更多地朝向前者——一个通用的智能体式协作者——调校,但所有这些行为都是相互重叠的。
At first glance, in aggregate, their model ranking definitely looks reasonable. Still, it shows signs of strain and confusion, as any benchmark will, with GPT 6.1 Terra beating Claude Fable 5.1 on AAv5. Today, some benchmarks measure the absolute frontier of capabilities, playing nicely in open-ended tasks through your favorite harness, which often does take a large model. Some other benchmarks measure certain skills that will be needed of a model in repetitive agentic tasks. I get the feeling that AA is tuning their benchmark more towards the former, a general agentic collaborator, but all of these behaviors overlap.
这让我们回到促使我写这篇文章的那张图,它展示了在新基准上开源模型与闭源模型的平衡。这张图显示,8 个闭源模型击败了最好的待开源权重模型 Kimi K3.1 Pro(未用蓝色标出)。该基准采取了一种试图衡量前沿模型发展方向的立场,这扩大了与开源模型的差距。这是一个可以接受的决定,但随着时间的推移,当开源模型的现有工作流成为经济的关键因素时,它可能会成为一个错过行业核心经济引擎所在的基准。
This brings us to the plot that prompted me to write this, which showed the balance of open vs. closed models on the new benchmark. This plot shows 8 closed models beating the best to-be open-weight model, Kimi K3.1 Pro (which isn’t colored blue). This benchmark takes the stance of trying to measure where frontier models are moving, which grows the gap to open models. This is an acceptable decision, but over time, as existing workflows of open models become key factors of the economy, it could become a benchmark that misses where the core economic engine of the industry is.
1. 将推理市场需求高的领域降低到硬件的商品价格。
1. Reducing domains with high market demand for inference to a commodity price of hardware.
2. 允许智能的所有权,将其专门化到内部数据,并在经济中的许多公司中控制内部智能体。
2. Allowing ownership of intelligence, specializing it to internal data, and controlling internal agents across many companies in the economy.
这两者都需要很长时间才能实现,并且在基准测试中只得到部分体现。当前算力供给如此之低,以至于开放模型提供商也能获得可观的利润率——当这一利润率收窄时,我们就会知道这套体系开始奏效了。向企业扩散也是同样的情况;由于如今构建评估和进行后训练需要困难的机器学习专业知识,其确切上限尚不明确。
Both of these will take a very long time to materialize and are partially represented in benchmarks. The supply of compute is currently so low that open model providers also are raking in solid margins – when this shrinks, we will know the system is starting to work. The same goes for diffusion into enterprises; it's not clear what the exact ceiling on it is because of the difficult ML expertise that is needed to build evaluations and do post-training today.
在下一个主要模型周期,比如 2-3 个月后随 GLM 6 或 Kimi 3.5 到来时,我预计领先的开放模型实验室将在这个指数上取得实质性进展,再次令许多观察者惊讶。这是一个我们已经持续多年的周期。
At the next major model cycle, say in 2-3 months with GLM 6 or Kimi 3.5, I expect the leading open model labs to make substantial steps on this index, again surprising many observers. This is a cycle we have been on for years now.
我已多次写到开放模型如何跟上前沿,总结如下:
I've written many times about how open models keep up with the frontier, and a summary is that:
1. 最好的开放模型会稍微针对基准测试做优化(benchmax)。最差的开放模型则大量针对基准测试做优化(但反正也得不到太多关注)。
1. The best open models do benchmax a little. The worst open models benchmax a lot (but don't get a ton of attention anyways).
2. 最好的开放模型所覆盖的能力范围略窄,如上文所示它们在科学领域存在差距。开放模型实验室正在努力解决这一问题,但我认为不聚焦于此可能会成为一个战略错误。
2. The best open models are a bit narrower in which capabilities they cover, as seen above with their gap on scientific domains. The open model labs are working to address this, but I think it could become a strategic error to not focus.
如今对性能差距的解释有一个更细致的版本,即训练中的许多收益来自于拥有分布接近评估的强化学习环境(更长的分析见此处)。要在多个领域保持快速进展,需要一个复杂的强化学习技术栈,并非所有开放模型实验室都具备。有些可能还依赖于购买具有中等交付周期的数据。
There’s a more nuanced version of explanations of the performance gap today, where so many of the gains in training are coming from having RL environments with distributions close to the evaluations (a longer analysis on this is here). Keeping up with rapid progress across multiple domains requires a sophisticated RL stack that not all the open model labs may possess. Some may be reliant on buying data that has moderate lead times as well.
该公告的另一部分是转向“私有基准”,即提示和测试环境不公开,这标志着开放实验室正在“刷榜”(benchmaxxing)。这类分析以前做过,符合我上面关于开放实验室如何刷榜的理论。当你知道评估的形状时,创建支持评估的训练数据更容易,但用隐藏数据也并非不可能。总之,这一变化在方向上我认为是站得住脚的,但并非一致同意。例如,你完全信任私有基准构建者,因为你不再受益于社区发现基准中的问题并推动基准自我改进——这种情况经常发生!涉及的资金足够多,我更希望看到基准测试的透明度,让众多眼睛发现漏洞。
Another part of this announcement is shifting to “private benchmarks”, where the prompts and test environments are not public, as a sign that open labs are benchmaxxing. This sort of analysis has been done before, and fits with my above theory on how open labs benchmax. It is easier to create training data to support an evaluation when you know the shape of it, but the task isn’t impossible with hidden data. All in, this is a change that directionally I think is defensible, but it’s not unanimous. For example, you’re putting complete trust in the private benchmark builder, as you no longer get the benefit of the community finding issues in the benchmark and pushing the benchmark to improve itself – which happens _a lot!_ There is enough money on the line, where I’d rather see transparency in benchmarking and enabling many eyes to find bugs.
我需要提出的最后一点,确实抬高了闭源实验室,即 Anthropic 和 OpenAI 等公司几个月甚至几年前就不再关心这些指数了。他们的重点绝大部分在于支持内部用途——如 AI 研发自动化——以及花费数十亿美元的客户的用例。是的,他们有团队成员在攀登其他能力以解锁新市场,但这些行为在较小程度上受公共基准驱动。同时,对中小型实验室来说,在这些智能指数上表现稳健是生存攸关的,因为他们的品牌完全建立在追赶前沿之上。有时这是一场非常愚蠢的猫鼠游戏,但将持续下去。
The final point I need to make, which does flatter the closed labs, is that the likes of Anthropic and OpenAI moved on from caring about these indices months, or even years, ago. Their focus is to a large majority on supporting internal uses – like AI R&D automation – and the use-cases of customers who are spending billions of dollars. Yes, they have team members hill climbing on other capabilities to unlock new markets, but these behaviors are to a minority extent driven by public benchmarks. At the same time, it is existential for small to medium sized labs to appear solid on these intelligence indices, as their branding is fully built around catching up to the frontier. It’s a very silly cat and mouse game at times, but one that will be lasting.
我们当前均衡——一两个指数主导着关于“前沿在哪里”的讨论——的唯一解决方案是拥有更多带有其他偏见的智能指数,例如仅针对编码(如 CursorBench)。在某些方面,这正在形成,Vals AI 和 Intelligence.ai 等正在成长,但注意力严重偏向 Artificial Analysis。
The only solution to our current equilibrium with one or two indices dominating the discourse on “where the frontier is” is to have more intelligence indices with other biases, e.g. just for coding (like CursorBench). In some ways this is forming, with the likes of Vals AI and Intelligence.ai growing, but the mindshare is very lopsided towards Artificial Analysis.
每个基准创建者都有自己的世界观。困境在于,你总是可以调整数字来匹配你的叙述。这一直是做好科学的挑战。这个问题最常见的版本是“你应该投入多少精力来调整训练基线?”更多的调优使你的结果更可信,但通常会降低你方法的表面成功率。在智能体时代,这一点加倍正确,因为许多分析工作是通过 LLM 的视角完成的。在评估中,这归结为你选择将哪些基准纳入你的指数。纳入一个支持你叙述的低质量基准,而不是一个与之冲突的更严格的基准,是很容易的。
Every benchmark creator has their worldview. The dilemma is that you can always shift the numbers to match your narrative. This has always been a challenge of doing good science. The most recurring version of this question has been “how much effort should you put in to tune your training baseline?” More tuning makes your results more trustworthy, but often reduces the apparent success of your method. This is doubly true in the era of agents, where much of the analysis work is done through the lens of LLMs. In evaluation, this comes down to which benchmarks you choose to include in your index. It is easy to include a lower quality benchmark that supports your narrative, rather than a more rigorous one that conflicts.
基准测试行业归根结底取决于信任。我对 Artificial Analysis 有相当程度的信任,但并非对生态系统中任何一方都给予最高信任。Artificial Analysis 的商业模式是从实验室那里赚钱来验证其结果。他们借此赚了很多钱,这无可厚非。其他基准测试公司也有类似问题,例如 Arena,它通过为实验室做发布前测试来支付账单。
The benchmarking industry comes down to trust. I have a solid amount of trust for Artificial Analysis, but not the most I do for anyone in the ecosystem. Artificial Analysis's business model is to make money from the labs to verify their results. They make a lot of money doing this, which is fine. Other benchmarking companies have a similar problem, such as Arena, which does pre-release testing for labs to pay the bills.
在评估的**公平性**上,我最信任的实体是非营利组织、学术界以及像 CAISI 这样的政府机构。如今,在前沿评估的**准确性**上,我信任的是 Artificial Analysis 和 Arena 这样的初创公司。因此,我们都困在权衡与两难之中。没有哪个数字是完美的,数字中存在大量回旋余地,人们可以据此把分析引向不同方向。这是第二个两难,某种程度上正是我的工作——作为基准测试者的基准测试者所面临的两难。它更偏社会学,而非科学。
The entities I trust the most to be _fair_ on evaluations are non-profits, academics, and government institutions like CAISI. Today, the institutions I trust to be _accurate_ on the frontier evaluation are startups like Artificial Analysis and Arena. So, we are all stuck with trade-offs and dilemmas. No number is perfect and there is a large amount of wiggle room in the numbers for people to take analysis in different directions. This is a second dilemma, one that is sort of my job, the dilemma of a benchmarker of benchmarkers. It is more sociological than scientific.
_感谢读到最后!我想向我的读者表达一份谢意。Interconnects 的成长很大程度上依赖读者的口口相传,所以如果你有朋友可能会喜欢这篇文章,请分享给他们。_
_Thanks for reading to the end! I wanted to add a note of appreciation to my audience. Much of Interconnects' growth relies on word of mouth from readers, so if you have a friend you think would like this, please share it._
_我还在 YouTube 上刚刚发布了一场关于开放模型的综合性入门讲座,你或许会喜欢。_
_I also just published a comprehensive introductory lecture on open models on YouTube, which you may enjoy._