An open question these days is why large language models work so well. In this blog post I will discuss six basic intuitions about large language models. Many of them are inspired by manually examining data, which is an exercise that I’ve found helpful and would recommend. Language models are pre-trained to simply predict the next word in a corpus of text, and they learn a surprising amount from this. Let’s look at some examples of what they might learn from this next-word prediction task. Intuition 1. Next-word prediction on large, self-supervised data is massively multi-task learning.
核心贡献 · Key contributions
在大型自监督数据上的下一个词预测相当于大规模多任务学习，隐式获得多种技能。 Next-token prediction on large self-supervised data acts as massively multi-task learning, acquiring diverse skills implicitly.
通过上下文学习，可将机器学习任务表述为下一个词预测，从而实现少样本提示。 Machine learning tasks can be framed as next-token prediction via in-context learning, enabling few-shot prompting.
词元信息密度不同；思维链提示为模型处理困难词元提供额外算力。 Tokens vary in information density; chain-of-thought prompting gives models extra compute for hard tokens.
根据缩放定律，扩展语言模型的规模和数据可预测地改善损失，降低训练不确定性。 Scaling language model size and data improves loss predictably according to scaling laws, reducing uncertainty in training.
虽然总体损失平滑缩放，但个体任务在规模超过阈值时可能出现涌现能力。 While overall loss scales smoothly, individual tasks may exhibit emergent abilities above a size threshold.
输入输出映射的真正上下文学习仅在足够大的语言模型中出现，小模型不具备。 Real in-context learning of input-output mappings emerges only in sufficiently large language models, not small ones.
局限 · Limitations
缩放为何有效仍是悬而未决的问题；解释较为模糊且未经证实。 The exact reasons why scaling works remain an open question; explanations are hand-wavy and unverified.
涌现能力无法通过从小模型外推缩放曲线预测，增加了预估难度。 Emergent abilities cannot be predicted by extrapolating scaling curves from smaller models, complicating forecasts.
上下文学习可能部分反映格式或标签线索而非真实的输入输出映射，尤其在 GPT-3 中。 In-context learning may partly reflect formatting or label cues rather than genuine input-output mapping, especially in GPT-3.
随机标签实验表明小模型忽略翻转标签，说明较小规模下真正上下文学习有限。 Random-label experiments show small models ignore flipped labels, indicating limited true in-context learning at smaller scales.
通过思维链让模型推理会增加推理算力，对复杂任务可能代价高昂。 Requiring models to reason via chain-of-thought adds inference compute, which may be costly for complex tasks.
论文章节 · Sections（共 2）
关于大型语言模型的六个直觉Six intuitions about large language models