High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm
核心贡献 · Key contributions
识别 LLM 服务中的内存分配挑战并量化其对性能的影响。 Identifies memory allocation challenges in LLM serving and quantifies their impact on performance.
提出 PagedAttention，一种在非连续分页内存上操作 KV 缓存的注意力算法。 Proposes PagedAttention, an attention algorithm operating on non-contiguous paged memory for KV cache.
设计并实现 vLLM，一个基于 PagedAttention 的分布式 LLM 服务引擎。 Designs and implements vLLM, a distributed LLM serving engine built on PagedAttention.
通过块级管理和按需分配实现 KV 缓存内存的近乎零浪费。 Achieves near-zero waste in KV cache memory via block-level management and on-demand allocation.
支持请求内部和请求之间的灵活 KV 缓存共享，适用于复杂解码算法。 Enables flexible KV cache sharing within and across requests for complex decoding algorithms.
相比 FasterTransformer 和 Orca 等最先进系统，吞吐量提升 2-4 倍。 Demonstrates 2-4x throughput improvement over state-of-the-art systems like FasterTransformer and Orca.
当系统受算力限制而非内存限制时，性能提升不那么显著。 Performance gains are less pronounced when the system is compute-bound rather than memory-bound.
块大小选择涉及 GPU 利用率和碎片化之间的权衡；默认值 16 可能不适用于所有工作负载。 Block size choice involves trade-off between GPU utilization and fragmentation; default 16 may not suit all workloads.
交换恢复方法在小块大小时因大量小数据传输而产生开销。 Swapping recovery method incurs overhead with small block sizes due to numerous small data transfers.
这些技术特定于 LLM 服务，可能不适用于具有静态内存模式的其他 GPU 工作负载。 Techniques are specific to LLM serving and may not benefit other GPU workloads with static memory patterns.
论文章节 · Sections（共 32）
摘要Abstract
1. 引言1. Introduction
2. 背景2. Background
2.1. 基于 Transformer 的大语言模型2.1. Transformer-Based Large Language Models
2.2. LLM 服务与自回归生成2.2. LLM Service & Autoregressive Generation
2.3. LLM 批处理技术2.3. Batching Techniques for LLMs
3. LLM 服务中的内存挑战3. Memory Challenges in LLM Serving
3.1. 现有系统中的内存管理3.1. Memory Management in Existing Systems
4. 方法4. Method
4.1. PagedAttention4.1. PagedAttention
4.2. KV 缓存管理器4.2. KV Cache Manager
4.3. 使用 PagedAttention 和 vLLM 解码4.3. Decoding with PagedAttention and vLLM
4.4. 在其他解码场景中的应用4.4. Application to Other Decoding Scenarios
4.5. 调度与抢占4.5. Scheduling and Preemption
4.6. 分布式执行4.6. Distributed Execution
5. 实现5. Implementation
5.1. 内核级优化5.1. Kernel-level Optimization
5.2. 支持多种解码算法5.2. Supporting Various Decoding Algorithms
6. 评估6. Evaluation
6.1. 实验设置6.1. Experimental Setup
6.2. 基本采样6.2. Basic Sampling
6.3. 并行采样与束搜索6.3. Parallel Sampling and Beam Search
6.4. 共享前缀6.4. Shared prefix
6.5. 聊天机器人6.5. Chatbot
7. 消融研究7. Ablation Studies
7.1. 内核微基准测试7.1. Kernel Microbenchmark
7.2. 块大小的影响7.2. Impact of Block Size
7.3. 重计算与交换的比较7.3. Comparing Recomputation and Swapping