The potential of Large Language Model (LLM) as agents has been widely acknowledged recently. Thus, there is an urgent need to quantitatively \textit{evaluate LLMs as agents} on challenging tasks in interactive environments. We present AgentBench, a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities. Our extensive test over \num API-based and open-sourced (OSS) LLMs shows that, while top commercial LLMs present a strong ability of acting as agents in complex environments, there is a significant disparity in performance between them and many OSS competitors that are no larger than 70B. We identify the typical reasons of failures in environments and LLMs, showing that poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles for developing usable LLM agents. Improving instruction following and training on high quality multi-round alignment data could improve agent performance. And different from existing assumptions, training on code present ambivalent impacts on different agent tasks. Datasets, environments, and an integrated evaluation package for AgentBench are released at https://github.com/THUDM/AgentBench.
核心贡献 · Key contributions
提出 AgentBench，一个包含 8 个环境的多维度基准，用于评估作为智能体的大语言模型。 Introduces AgentBench, a multi-dimensional benchmark with 8 environments for evaluating LLMs as agents.
评估了 27 个大语言模型，揭示了顶级商业模型与开源模型之间的显著性能差距。 Evaluates 27 LLMs, revealing significant performance gap between top commercial and open-source models.
指出长期推理、决策和指令遵循能力不足是开发可用大语言模型智能体的主要障碍。 Identifies poor long-term reasoning, decision-making, and instruction following as main obstacles for LLM agents.
表明代码训练和高质量对齐数据可以提升智能体性能。 Shows code training and high-quality alignment data can improve agent performance.
发布了 AgentBench 的数据集、环境和集成评估工具包。 Releases datasets, environments, and an integrated evaluation toolkit for AgentBench.
局限 · Limitations
仅评估纯文本大语言模型，不包括多模态模型。 Only evaluates text-only LLMs, excluding multi-modal models.
评估的开源大语言模型仅限于参数小于 70B 的模型。 Open-source LLMs evaluated are limited to those smaller than 70B parameters.
仅使用基本的思维链提示，未采用反思或搜索等高级策略。 Uses only basic Chain-of-Thought prompting, not advanced strategies like reflection or search.
由于环境简化，评估可能无法完全反映实际部署情况。 Evaluation may not fully reflect real-world deployment due to simplified environments.
分数归一化方法可能掩盖任务间的绝对性能差异。 Score normalization method may obscure absolute performance differences across tasks.
论文章节 · Sections（共 7）
摘要Abstract
1 引言1 Introduction
2 大语言模型作为智能体：定义与基础2 LLM-as-Agent: Definition and Preliminary
3 AgentBench 的构成：概览3 Composition of AgentBench: A Brief Look