Real-time speech interaction, serving as a fundamental interface for human-machine collaboration, holds immense potential. However, current open-source models face limitations such as high costs in voice data collection, weakness in dynamic control, and limited intelligence. To address these challenges, this paper introduces Step-Audio, the first production-ready open-source solution. Key contributions include: 1) a 130B-parameter unified speech-text multi-modal model that achieves unified understanding and generation, with the Step-Audio-Chat version open-sourced; 2) a generative speech data engine that establishes an affordable voice cloning framework and produces the open-sourced lightweight Step-Audio-TTS-3B model through distillation; 3) an instruction-driven fine control system enabling dynamic adjustments across dialects, emotions, singing, and RAP; 4) an enhanced cognitive architecture augmented with tool calling and role-playing abilities to manage complex tasks effectively. Based on our new StepEval-Audio-360 evaluation benchmark, Step-Audio achieves state-of-the-art performance in human evaluations, especially in terms of instruction following. On open-source benchmarks like LLaMA Question, shows 9.3% average performance improvement, demonstrating our commitment to advancing the development of open-source multi-modal language technologies. Our code and models are available at https://github.com/stepfun-ai/Step-Audio.
核心贡献 · Key contributions
提出 Step-Audio，首个生产级开源统一语音-文本多模态模型，拥有 1300 亿参数。 Proposes Step-Audio, the first production-ready open-source unified speech-text multi-modal model with 130B parameters.
引入生成式语音数据引擎，实现低成本语音克隆，并通过蒸馏得到轻量级 Step-Audio-TTS-3B 模型。 Introduces a generative speech data engine for affordable voice cloning and distills a lightweight Step-Audio-TTS-3B model.
开发指令驱动的精细控制系统，支持方言、情感、歌唱和说唱的动态调整。 Develops an instruction-driven fine control system enabling dynamic adjustments across dialects, emotions, singing, and RAP.
增强认知架构，集成工具调用和角色扮演能力，以管理复杂任务。 Enhances cognitive architecture with tool calling and role-playing abilities for complex task management.
在全新 StepEval-Audio-360 基准测试中取得最先进性能，尤其在指令遵循方面。 Achieves state-of-the-art performance on the new StepEval-Audio-360 benchmark, especially in instruction following.
在 LLaMA Question 等开源基准上平均提升 9.3%，推动开源多模态语言技术发展。 Demonstrates 9.3% average improvement on open-source benchmarks like LLaMA Question, advancing open-source multi-modal language technologies.
局限 · Limitations
模型在纯语音对话（AQAA）上的性能尚未完全优化，依赖中间文本转换。 The model's performance on pure voice dialogue (AQAA) is not yet fully optimized, relying on intermediate text conversion.
合成数据引擎可能引入种子数据的偏差，需要仔细的质量控制。 The synthetic data engine may introduce biases from the seed data and requires careful quality control.
当前框架专注于语音-文本集成，尚未原生融入视觉模态。 The current framework focuses on speech-text integration and has not yet incorporated vision modality natively.
评估基准 StepEval-Audio-360 主要面向中文，限制了向其他语言的泛化能力。 The evaluation benchmark StepEval-Audio-360 is primarily in Chinese, limiting generalizability to other languages.
论文章节 · Sections（共 20）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 架构3 Architecture
3.1 分词器3.1 Tokenizer
3.2 大语言模型3.2 LLM
3.3 语音解码器3.3 Speech Decoder
3.4 实时推理3.4 Real-time Inference
4.1 数据集4.1 Dataset
4.2 训练细节4.2 Training Detail
4.3 训练基础设施4.3 Training Infrastructure
4.4 音频预训练分词器探索4.4 Exploring Tokenizer for Audio Pretraining