DeepSeek Sparse Attention (DSA) enables efficient long-context modeling through its Lightning Indexer. However, practical deployment remains constrained by the indexer's expensive $O(L^2)$ scoring overhead and the hardware-inefficient, discontinuous memory-access patterns induced by its outputs. To address these system-level bottlenecks, we introduce LongCat Sparse Attention (LSA), a hardware-algorithm co-designed framework comprising three complementary and orthogonal strategies: (1) Streaming-Aware Indexing, which selectively converts scattered KV entries into hardware-aligned contiguous layouts to enable coalesced HBM access; (2) Cross-Layer Indexing, which amortizes indexing overhead by reusing the results produced by a single layer across consecutive layers, supported by cross-layer distillation; and (3) Hierarchical Indexing, which adopts a coarse-to-fine scoring scheme to progressively narrow the candidate set for each query, thereby substantially reducing indexing computation. Extensive scaling experiments, ranging from 69B-A3B to 560B-A27B models, demonstrate that LSA consistently achieves performance on par with full attention across both general-purpose and long-context benchmarks. Moreover, LSA supports native training with context lengths of up to one million tokens and underpins the development of LongCat-2.0 (1.6T-A48B). To facilitate further research, we also introduce and open-source LongCat-Flash-Lite-Sparse (69B-A3B), which integrates LSA into LongCat-Flash-Lite and incorporates an updated long-context training corpus.
核心贡献 · Key contributions
提出 LongCat 稀疏注意力（LSA），一种硬件-算法协同设计的框架，包含三种正交策略：流感知索引、跨层索引和分层索引。 Proposes LongCat Sparse Attention (LSA), a hardware-algorithm co-designed framework with three orthogonal strategies: Streaming-Aware, Cross-Layer, and Hierarchical Indexing.
跨层索引通过跨层蒸馏支持跨层复用索引器输出，将索引计算量减少至 1/N。 Cross-Layer Indexing amortizes indexing overhead by reusing indexer outputs across layers, supported by cross-layer distillation, reducing indexing computation to 1/N.
分层索引采用由粗到细的评分方案，将每查询选择复杂度从 O(L)降至 O(L/P + MP)，在 1024K 长度下实现高达 4.11 倍加速。 Hierarchical Indexing uses a coarse-to-fine scoring scheme to reduce per-query selection complexity from O(L) to O(L/P + MP), with up to 4.11x speedup at 1024K.
LSA 在 69B-A3B 至 560B-A27B 规模上，在各类基准测试中与全注意力性能持平，并支持高达一百万 token 的原生训练。 LSA matches full attention performance across benchmarks at scales from 69B-A3B to 560B-A27B, and supports native training up to one million tokens.
发布 LongCat-Flash-Lite-Sparse（69B-A3B），一个集成 LSA 的开源模型，具有扩展的上下文长度和增强的智能体能力。 Releases LongCat-Flash-Lite-Sparse (69B-A3B), an open-source model integrating LSA with extended context length and improved agentic capabilities.
局限 · Limitations
LSA 减少了注意力计算，但并未减少总 KV 缓存占用，因为每个 token 仍需存储 KV 条目。 LSA reduces attention computation but does not reduce total KV-cache footprint, as every token still stores a KV entry.
分层索引仅在序列长度>=256K 时提供净效率提升；低于此长度，其开销超过收益。 Hierarchical Indexing provides net efficiency gains only for sequence lengths >= 256K; below this, its overhead outweighs benefits.
跨层索引在组大小 N=4 时在长上下文验证中导致可测量的精度损失，限制了复用深度。 Cross-Layer Indexing with group size N=4 causes measurable accuracy loss on long-context validation, limiting reuse depth.
流感知索引将 50%的预算分配给固定区域，可能在某些任务中限制动态选择的灵活性。 Streaming-Aware Indexing allocates 50% of budget to fixed regions, potentially limiting dynamic selection flexibility in some tasks.
LSA 的有效性依赖于跨层蒸馏和超参数（如页大小和召回预算）的仔细调整。 LSA's effectiveness relies on cross-layer distillation and careful tuning of hyperparameters like page size and recall budget.
论文章节 · Sections（共 28）
LongCat 稀疏注意力：通过流式感知的层级跨层索引驯服闪电LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
1 引言1 Introduction
2 稀疏注意力与 DSA 基线2 Sparse Attention and the DSA Baseline
2.1 从固定模式到基于检索的稀疏注意力2.1 From Fixed Patterns to Retrieval-Based Sparse Attention
2.2 DSA 机制简述2.2 A Brief Recapitulation of DSA Mechanism
2.3 DSA 效率瓶颈分析2.3 Profiling the Efficiency Bottlenecks of DSA