This paper presents Step-Audio 2, an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation. By integrating a latent audio encoder and reasoning-centric reinforcement learning (RL), Step-Audio 2 achieves promising performance in automatic speech recognition (ASR) and audio understanding. To facilitate genuine end-to-end speech conversation, Step-Audio 2 incorporates the generation of discrete audio tokens into language modeling, significantly enhancing its responsiveness to paralinguistic information such as speaking styles and emotions. To effectively leverage the rich textual and acoustic knowledge in real-world data, Step-Audio 2 integrates retrieval-augmented generation (RAG) and is able to call external tools such as web search to mitigate hallucination and audio search to switch timbres. Trained on millions of hours of speech and audio data, Step-Audio 2 delivers intelligence and expressiveness across diverse conversational scenarios. Evaluation results demonstrate that Step-Audio 2 achieves state-of-the-art performance on various audio understanding and conversational benchmarks compared to other open-source and commercial solutions. Please visit https://github.com/stepfun-ai/Step-Audio2 for more information.
核心贡献 · Key contributions
集成潜在音频编码器和以推理为中心的强化学习，增强音频理解能力。 Integrates latent audio encoder and reasoning-centric RL for enhanced audio understanding.
将离散音频令牌生成融入语言建模，实现端到端语音交互。 Incorporates discrete audio token generation into language modeling for end-to-end speech interaction.
集成检索增强生成和外部工具，如网络搜索和音频搜索。 Integrates retrieval-augmented generation and external tools like web and audio search.
采用多阶段训练策略，在 800 万小时语音和音频数据上训练。 Trained on 8 million hours of speech and audio data with multi-stage training strategy.
在 ASR、音频理解和语音对话基准上达到最先进性能。 Achieves state-of-the-art performance on ASR, audio understanding, and speech conversation benchmarks.
引入 StepEval-Audio-Paralinguistic 和 StepEval-Audio-Toolcall 基准。 Introduces StepEval-Audio-Paralinguistic and StepEval-Audio-Toolcall benchmarks.
局限 · Limitations
评估限于中文和英文，多语言结果部分覆盖。 Evaluation limited to Chinese and English, with partial multilingual results.
依赖合成数据进行副语言和工具调用训练。 Relies on synthetic data for paralinguistic and tool-call training.
工具调用评估仅使用中文语音对话。 Tool-call evaluation uses only Chinese speech conversations.
音频搜索工具需要预构建的语音库。 Audio search tool requires a pre-built voice library.
英文语音对话性能略低于 GPT-4o Audio。 Performance on English speech conversation slightly below GPT-4o Audio.
论文章节 · Sections（共 17）
摘要Abstract
1 引言1 Introduction
2.1 语音与音频理解2.1 Speech and audio understanding
2.2 文本到语音合成2.2 Text-to-speech synthesis
2.3 语音到语音翻译2.3 Speech-to-speech translation
2.4 语音到文本与语音到语音对话2.4 Speech-to-text and speech-to-speech conversation
3.1 架构3.1 Architecture
3.2 预训练3.2 Pre-training
3.3 有监督微调3.3 Supervised fine-tuning
3.4 强化学习3.4 Reinforcement learning
4.1 自动语音识别4.1 Automatic speech recognition
4.2 副语言信息理解4.2 Paralinguistic information understanding