Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.
核心贡献 · Key contributions
识别出注意力汇聚现象，即初始词元吸引高注意力分数，对窗口注意力性能至关重要。 Identifies attention sink phenomenon where initial tokens attract high attention scores, crucial for window attention performance.
提出 StreamingLLM 框架，使大语言模型无需微调即可泛化到无限序列长度。 Proposes StreamingLLM framework enabling LLMs to generalize to infinite sequence lengths without fine-tuning.
证明在预训练期间添加占位符词元作为专用注意力汇聚可改善流式部署。 Demonstrates that adding a placeholder token as a dedicated attention sink during pre-training improves streaming deployment.
在流式设置中，相比滑动窗口重计算基线实现高达 22.2 倍加速。 Achieves up to 22.2x speedup over sliding window recomputation baseline in streaming settings.
验证了四个初始词元作为注意力汇聚足以在多个模型家族中实现稳定流式性能。 Validates that four initial tokens suffice as attention sinks for stable streaming performance across multiple model families.
表明使用汇聚词元预训练不会损害收敛性或下游任务准确率。 Shows that pre-training with a sink token does not harm convergence or downstream task accuracy.
局限 · Limitations
StreamingLLM 不扩展注意力窗口大小或改善长上下文利用。 StreamingLLM does not expand the attention window size or improve long-context utilization.
增加缓存大小并不一致降低困惑度，表明上下文利用有限。 Increasing cache size does not consistently lower perplexity, indicating limited context utilization.
该方法依赖相对位置编码；绝对位置编码可能需要调整。 The method relies on relative positional encoding; absolute positional encoding may require adaptation.
注意力汇聚现象可能不泛化到非自回归模型或仅编码器架构。 Attention sink phenomenon may not generalize to non-autoregressive models or encoder-only architectures.
当初始词元数量不足（如一或两个）时，StreamingLLM 性能下降。 StreamingLLM's performance degrades when the number of initial tokens is insufficient (e.g., one or two).
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 StreamingLLM3 StreamingLLM
3.1 窗口注意力与注意力汇聚的失效3.1 The Failure of Window Attention and Attention Sinks
3.2 带注意力汇聚的滚动 KV 缓存3.2 Rolling KV Cache with Attention Sinks
3.3 使用注意力汇聚预训练 LLM3.3 Pre-Training LLMs with Attention Sinks
4 实验4 Experiments
4.1 跨 LLM 系列和规模的长文本语言建模4.1 Language Modeling on Long Texts Across LLM Families and Scales
4.2 使用汇聚令牌预训练的结果4.2 Results of Pre-Training with a Sink Token
4.3 指令调优模型的流式问答结果4.3 Results on Streaming Question Answering with Instruction-tuned Models