Large language models (LLMs) are increasingly used for complex tasks that require multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs. However, efficient systems are lacking for programming and executing these applications. We introduce SGLang, a system for efficient execution of complex language model programs. SGLang consists of a frontend language and a runtime. The frontend simplifies programming with primitives for generation and parallelism control. The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding. Experiments show that SGLang achieves up to 6.4x higher throughput compared to state-of-the-art inference systems on various large language and multi-modal models on tasks including agent control, logical reasoning, few-shot learning benchmarks, JSON decoding, retrieval-augmented generation pipelines, and multi-turn chat. The code is publicly available at https://github.com/sgl-project/sglang
核心贡献 · Key contributions
提出 SGLang，一种具有 LLM 编程原语的领域特定语言，支持并行、控制流、嵌套调用和外部调用。 Proposes SGLang, a domain-specific language with primitives for LLM programming, supporting parallelism, control flow, nested calls, and external calls.
开发了一个与 Python 协同执行 SGLang 程序的解释器，将提示状态作为异步流管理，实现程序内并行。 Develops an interpreter that executes SGLang programs with Python, managing prompt states as asynchronous streams for intra-program parallelism.
构建了一个编译器，将 SGLang 程序编译为计算图，支持代码移动和预取等经典编译器优化。 Builds a compiler that compiles SGLang programs into computational graphs, enabling classical compiler optimizations like code movement and prefetching.
引入 RadixAttention，一种通过带 LRU 淘汰的基数树在多次生成调用间自动复用 KV 缓存的新技术。 Introduces RadixAttention, a novel technique for automatic KV cache reuse across multiple generation calls using a radix tree with LRU eviction.
在包括智能体、推理和 JSON 解码等多种 LLM 任务上，相比最先进的推理系统实现了高达 6.4 倍的吞吐量提升。 Demonstrates up to 6.4x throughput improvement over state-of-the-art inference systems on diverse LLM tasks including agents, reasoning, and JSON decoding.
在复杂提示流程中，相比 OpenAI API，代码行数减少 55%，展示了生产力提升。 Shows productivity gains with 55% fewer lines of code compared to OpenAI APIs for complex prompting flows.
局限 · Limitations
追踪和编译器缺乏对数据依赖控制流的支持，将编译限制在没有动态分支的程序上。 Tracing and compiler lack support for data-dependent control flows, limiting compilation to programs without dynamic branches.
当前分词算法可能在提示和生成边界产生伪影，需要令牌修复来解决。 Current tokenization algorithm may create artifacts at prompt and generation boundaries, requiring token healing for resolution.
尚未实现语法约束解码，需要未来集成高效技术和仔细的批处理。 Grammar-constrained decoding is not yet implemented, requiring future integration of efficient techniques and careful batching.
系统尚未在超大模型（如 100B+参数）或分布式设置上测试，限制了可扩展性声明。 The system has not been tested on extremely large models (e.g., 100B+ parameters) or distributed settings, limiting scalability claims.
使用 GPT-4 进行代码移动的编译器优化并非完全可靠，在某些情况下无法正确理解语义。 Compiler optimizations using GPT-4 for code movement are not fully reliable, with failures in understanding semantic meaning in some cases.
论文章节 · Sections（共 22）
摘要Abstract
1 引言1 Introduction
2.1 大语言模型的推理过程2.1 The Inference Process of LLMs
2.2 编程大语言模型的范式2.2 The Paradigm of Programming LLMs
2.3 挑战与机遇2.3 Challenges and Opportunities
2.4 现有的大语言模型编程系统2.4 Existing LLM Programming Systems
3 概述3 Overview
4.1 语法与原语4.1 Syntax and Primitives
4.2 解释器4.2 Interpreter
4.3 编译器4.3 Compiler
5 基于 RadixAttention 的运行时5 Runtime with RadixAttention