Kimi-Audio 技术报告

Kimi-Audio Technical Report

杨植麟 Zhilin Yang · Moonshot AI · 2025-04-25 · arXiv:2504.18425 ↗ · 被引 210

摘要 · Abstract

我们介绍了 Kimi-Audio，一个在音频理解、生成和对话方面表现出色的开源音频基础模型。我们详细阐述了构建 Kimi-Audio 的实践，包括模型架构、数据整理、训练方案、推理部署和评估。具体来说，我们利用了一个 12.5Hz 的音频分词器，设计了一种新颖的基于 LLM 的架构，以连续特征作为输入、离散标记作为输出，并开发了一种基于流匹配的块式流式解分词器。我们整理了一个包含超过 1300 万小时音频数据的预训练数据集，涵盖语音、声音和音乐等多种模态，并构建了一个用于构建高质量、多样化后训练数据的流程。Kimi-Audio 从预训练的 LLM 初始化，通过几个精心设计的任务在音频和文本数据上持续预训练，然后进行微调以支持多种音频相关任务。大量评估表明，Kimi-Audio 在多项音频基准测试中取得了最先进的性能，包括语音识别、音频理解、音频问答和语音对话。我们在 https://github.com/MoonshotAI/Kimi-Audio 上发布了代码、模型检查点和评估工具包。

We present Kimi-Audio, an open-source audio foundation model that excels in audio understanding, generation, and conversation. We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation. Specifically, we leverage a 12.5Hz audio tokenizer, design a novel LLM-based architecture with continuous features as input and discrete tokens as output, and develop a chunk-wise streaming detokenizer based on flow matching. We curate a pre-training dataset that consists of more than 13 million hours of audio data covering a wide range of modalities including speech, sound, and music, and build a pipeline to construct high-quality and diverse post-training data. Initialized from a pre-trained LLM, Kimi-Audio is continual pre-trained on both audio and text data with several carefully designed tasks, and then fine-tuned to support a diverse of audio-related tasks. Extensive evaluation shows that Kimi-Audio achieves state-of-the-art performance on a range of audio benchmarks including speech recognition, audio understanding, audio question answering, and speech conversation. We release the codes, model checkpoints, as well as the evaluation toolkits in https://github.com/MoonshotAI/Kimi-Audio.

核心贡献 · Key contributions

提出了一种新颖的基于 LLM 的架构，以连续特征作为输入、离散标记作为输出，用于音频处理。
Proposes a novel LLM-based architecture with continuous features as input and discrete tokens as output for audio.
整理了超过 1300 万小时的音频预训练数据集，涵盖语音、声音和音乐。
Curates a pre-training dataset of over 13 million hours of audio covering speech, sound, and music.
开发了基于流匹配的逐块流式解标记器，用于低延迟语音生成。
Develops a chunk-wise streaming detokenizer based on flow matching for low-latency speech generation.
在语音识别、音频理解、音频问答和语音对话基准测试中取得了最先进的性能。
Achieves state-of-the-art performance on speech recognition, audio understanding, audio question answering, and speech conversation benchmarks.
向社区开源了代码、模型检查点和评估工具包。
Releases open-source codes, model checkpoints, and evaluation toolkits to the community.

局限 · Limitations

预训练依赖 ASR 转录，忽略了副语言和环境音频信息。
Pre-training relies on ASR transcription, neglecting paralinguistic and environmental audio information.
音频表示（语义标记）侧重于转录，缺少丰富的声学细节。
Audio representations (semantic tokens) focus on transcription, missing rich acoustic details.
严重依赖 ASR 和 TTS 生成训练数据，限制了性能上限。
Heavy dependence on ASR and TTS for training data, limiting performance ceiling.
模型主要在英语和普通话上评估，对其他语言的泛化能力有限。
Model evaluated mainly on English and Mandarin; generalization to other languages is limited.

论文章节 · Sections（共 20）

摘要 Abstract
1 引言 1 Introduction
2 架构 2 Architecture
2.1 概述 2.1 Overview
2.2 音频分词器 2.2 Audio Tokenizer
2.3 音频大语言模型 2.3 Audio LLM
2.4 音频解分词器 2.4 Audio Detokenizer
3.1 预训练数据 3.1 Pre-Training Data
3.2 监督微调数据 3.2 SFT Data
4.1 预训练 4.1 Pre-training
4.2 监督微调 4.2 Supervised Fine-tuning
4.3 音频解分词器训练 4.3 Training of Audio Detokenizer
5 推理与部署 5 Inference and Deployment
5.1 实时语音对话工作流 5.1 Workflow of Real-Time Speech Conversation
5.2 生产部署 5.2 Production Deployment
6 评估 6 Evaluation
6.1 评估工具包 6.1 Evaluation Toolkit
6.2 评估结果 6.2 Evaluation Results
7 相关工作 7 Related Work
8 挑战与未来趋势 8 Challenges and Future Trends

阅读逐段中英对照全文 →