语言模型（LM）在仅凭少量示例或文本指令解决新任务方面表现出色，尤其是在规模扩大时。然而，它们却在算术或事实查找等基本功能上表现不佳，而这些功能更简单、更小的模型却擅长。在本文中，我们展示了 LM 可以通过简单的 API 自学使用外部工具，从而两全其美。我们引入了 Toolformer，这是一个经过训练的模型，能够决定调用哪些 API、何时调用、传递什么参数以及如何最好地将结果整合到未来的令牌预测中。这是以自监督方式完成的，每个 API 仅需少量演示。我们整合了一系列工具，包括计算器、问答系统、两种不同的搜索引擎、翻译系统和日历。Toolformer 在多种下游任务中实现了显著改进的零样本性能，通常与更大的模型竞争，且不牺牲其核心语言建模能力。
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller models excel. In this paper, we show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. This is done in a self-supervised way, requiring nothing more than a handful of demonstrations for each API. We incorporate a range of tools, including a calculator, a Q\&A system, two different search engines, a translation system, and a calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.
核心贡献 · Key contributions
提出 Toolformer，一种自监督方法，让语言模型学习何时以及如何通过 API 调用使用外部工具。 Proposes Toolformer, a self-supervised method for LMs to learn when and how to use external tools via API calls.
使用基于困惑度的过滤方法，从语言模型生成的候选中选择有用的 API 调用。 Uses perplexity-based filtering to select helpful API calls from LM-generated candidates.
在算术、事实查找、问答和时间任务上取得强大的零样本性能，常超越更大模型。 Achieves strong zero-shot performance on arithmetic, factual lookup, QA, and temporal tasks, often beating much larger models.
保留核心语言建模能力，标准基准上的困惑度不受影响。 Preserves core language modeling abilities, as perplexity on standard benchmarks remains unaffected.
展示规模扩张的好处：更大的语言模型从学习工具使用中获益更多。 Demonstrates scaling benefits: larger LMs benefit more from learning tool use.
局限 · Limitations
无法链式调用多个 API；每个调用独立生成。 Cannot chain multiple API calls; each call is generated independently.