This paper introduces MiniCPM4, a highly efficient large language model (LLM) designed explicitly for end-side devices. We achieve this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems. Specifically, in terms of model architecture, we propose InfLLM v2, a trainable sparse attention mechanism that accelerates both prefilling and decoding phases for long-context processing. Regarding training data, we propose UltraClean, an efficient and accurate pre-training data filtering and generation strategy, and UltraChat v2, a comprehensive supervised fine-tuning dataset. These datasets enable satisfactory model performance to be achieved using just 8 trillion training tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient pre-training strategy search, and improve existing post-training methods by introducing chunk-wise rollout for load-balanced reinforcement learning and data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose CPM.cu that integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding. To meet diverse on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B parameters, respectively. Furthermore, we construct a hybrid reasoning model, MiniCPM4.1, which can be used in both deep reasoning mode and non-reasoning mode. Evaluation results demonstrate that MiniCPM4 and MiniCPM4.1 outperform similar-sized open-source models across benchmarks, with the 8B variants showing significant speed improvements on long sequence understanding and generation.
核心贡献 · Key contributions
提出 InfLLM v2，一种可训练的稀疏注意力机制，加速长上下文处理的预填充和解码阶段。 Proposes InfLLM v2, a trainable sparse attention mechanism accelerating both prefilling and decoding for long-context processing.
引入 UltraClean 高效预训练数据过滤策略和 UltraChat v2 高质量推理密集型数据生成方法。 Introduces UltraClean, an efficient pre-training data filtering strategy, and UltraChat v2 for high-quality reasoning-intensive data generation.
开发 ModelTunnel v2 实现高效训练策略搜索，以及分块展开实现负载均衡的强化学习。 Develops ModelTunnel v2 for efficient training strategy search and chunk-wise rollout for load-balanced reinforcement learning.
提出 CPM.cu 轻量级 CUDA 推理框架，集成稀疏注意力、量化和推测采样。 Presents CPM.cu, a lightweight CUDA inference framework integrating sparse attention, quantization, and speculative sampling.
仅用 Qwen3-8B 22% 的预训练词元即达到可比性能，在 128K 长度文档上实现 7 倍加速。 Achieves comparable performance to Qwen3-8B using only 22% of its pre-training tokens, with 7x speedup on 128K-length documents.
构建 MiniCPM4.1 混合推理模型，支持深度推理和非推理两种模式。 Constructs MiniCPM4.1 hybrid reasoning model supporting both deep and non-reasoning modes.
局限 · Limitations
稀疏注意力效率依赖 GPU 张量核心约束，每个查询组至少需要 16 个头。 Sparse attention efficiency relies on GPU tensor core constraints, requiring at least 16 heads per query group.
数据过滤和生成流程仍部分依赖人工设计的启发式规则。 Data filtering and generation pipelines still partially depend on human-designed heuristics.
多词元预测和 FP8 训练需要特定硬件支持，可能无法泛化到所有设备。 Multi-token prediction and FP8 training require specific hardware support and may not generalize to all devices.
评估主要基于英文和中文基准，未探索多语言和多模态场景。 Evaluation primarily on English and Chinese benchmarks; multilingual and multimodal settings are not explored.
三值 LLM BitCPM4 结果有前景，但在资源极度受限设备上可能存在性能下降。 Ternary LLM BitCPM4 shows promising results but may have performance degradation on extremely resource-limited devices.
论文章节 · Sections（共 23）
摘要Abstract
1 引言1 Introduction
2 高效架构与预训练2 Efficient Architecture and Pre-training
2.1 InfLLM v2：面向预填充与解码的可训练稀疏注意力2.1 InfLLM v2: Trainable Sparse Attention for Prefilling and Decoding
2.2 UltraClean：高质量预训练数据过滤与生成2.2 UltraClean: High-Quality Pre-Training Data Filtering and Generation