Qwen2-Audio 技术报告

Qwen2-Audio Technical Report

林俊旸 Junyang Lin · Alibaba Qwen · 2024-07-15 · arXiv:2407.10759 ↗ · 被引 642

摘要 · Abstract

我们介绍了 Qwen-Audio 的最新进展，即名为 Qwen2-Audio 的大规模音频语言模型，它能够接受各种音频信号输入，并根据语音指令进行音频分析或直接生成文本回复。与复杂的层次化标签不同，我们通过使用自然语言提示来简化不同数据和任务的预训练过程，并进一步扩展了数据量。我们增强了 Qwen2-Audio 的指令跟随能力，并实现了两种不同的音频交互模式：语音聊天和音频分析。在语音聊天模式下，用户可以无需文本输入，自由地与 Qwen2-Audio 进行语音交互。在音频分析模式下，用户可以在交互过程中提供音频和文本指令进行分析。注意，我们未使用任何系统提示来切换语音聊天和音频分析模式。Qwen2-Audio 能够智能地理解音频内容，并遵循语音命令做出适当响应。例如，在同时包含声音、多说话人对话和语音命令的音频片段中，Qwen2-Audio 可以直接理解命令并提供对音频的解释和响应。此外，DPO 优化了模型在事实性和遵循期望行为方面的性能。根据 AIR-Bench 的评估结果，Qwen2-Audio 在专注于音频中心指令跟随能力的测试中优于之前的 SOTA，如 Gemini-1.5-pro。Qwen2-Audio 已开源，旨在推动多模态语言社区的发展。

We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community.

核心贡献 · Key contributions

使用自然语言提示代替分层标签简化了预训练过程。
Simplified pre-training using natural language prompts instead of hierarchical tags.
引入了两种不同的交互模式：语音聊天和音频分析，无需系统提示。
Introduced two distinct interaction modes: voice chat and audio analysis, without system prompts.
应用 DPO 优化了事实性和对期望行为的遵循。
Applied DPO to optimize factuality and adherence to desired behavior.
在 AIR-Bench 聊天基准测试中取得了最先进的结果，优于 Gemini-1.5-pro。
Achieved state-of-the-art results on AIR-Bench chat benchmark, outperforming Gemini-1.5-pro.
开源模型以促进多模态语言社区的发展。
Open-sourced the model to foster multi-modal language community advancement.
扩大了训练数据规模并提升了指令遵循能力。
Scaled up training data and improved instruction-following capability.

局限 · Limitations

在某些数据集（如 Common Voice 15）上的评估并非零样本。
Evaluation on some datasets like Common Voice 15 was not zero-shot.
Gemini-1.5 在 AIR-Bench 上的结果因安全过滤而减少。
Gemini-1.5 results on AIR-Bench were reduced due to safety filtering.
模型在训练未覆盖的任务上性能可能下降。
Model performance may degrade on tasks not covered in training.
音频编码器从 Whisper-large-v3 初始化可能继承其偏差。
Audio encoder initialized from Whisper-large-v3 may inherit its biases.
对真实世界嘈杂音频场景的评估有限。
Limited evaluation on real-world noisy audio scenarios.

论文章节 · Sections（共 8）

摘要 Abstract
引言 Introduction
方法论 Methodology
3.1 评估 3.1 Evaluation
3.2 主要结果 3.2 Main Results
案例 Cases
结论 Conclusion
致谢 Acknowledgements

阅读逐段中英对照全文 →