Qwen3-Omni 技术报告

Qwen3-Omni Technical Report

林俊旸 Junyang Lin · Alibaba Qwen · 2025-09-22 · arXiv:2509.17765 ↗ · 被引 355

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

我们提出了 Qwen3-Omni，这是一个单一多模态模型，首次在文本、图像、音频和视频上均保持最先进性能，且相对于单模态对应模型没有任何退化。Qwen3-Omni 在 Qwen 系列中与同等大小的单模态模型性能相当，尤其在音频任务上表现出色。在 36 个音频和音视频基准测试中，Qwen3-Omni 在 32 个基准上达到开源 SOTA，在 22 个上达到整体 SOTA，超越了 Gemini-2.5-Pro、Seed-ASR 和 GPT-4o-Transcribe 等强大的闭源模型。Qwen3-Omni 采用 Thinker-Talker MoE 架构，统一了文本、图像、音频和视频的感知与生成，产生流畅的文本和自然的实时语音。它支持 119 种语言的文本交互、19 种语言的语音理解和 10 种语言的语音生成。为了降低流式合成中的首包延迟，Talker 使用多码本方案自回归预测离散语音编解码器。利用这些码本的表示能力，我们用轻量级因果卷积网络替代了计算密集的分块扩散，实现了从第一个编解码帧开始的流式传输。在冷启动设置下，Qwen3-Omni 实现了 234 毫秒的理论端到端首包延迟。为了进一步增强多模态推理，我们引入了一个 Thinking 模型，该模型对任何模态的输入进行显式推理。由于研究社区目前缺乏通用的音频字幕模型，我们对 Qwen3-Omni-30B-A3B 进行了微调，得到了 Qwen3-Omni-30B-A3B-Captioner，该模型能为任意音频输入生成详细且低幻觉的字幕。Qwen3-Omni-30B-A3B、Qwen3-Omni-30B-A3B-Thinking 和 Qwen3-Omni-30B-A3B-Captioner 已在 Apache 2.0 许可下公开发布。

We present Qwen3-Omni, a single multimodal model that, for the first time, maintains state-of-the-art performance across text, image, audio, and video without any degradation relative to single-modal counterparts. Qwen3-Omni matches the performance of same-sized single-modal models within the Qwen series and excels particularly on audio tasks. Across 36 audio and audio-visual benchmarks, Qwen3-Omni achieves open-source SOTA on 32 benchmarks and overall SOTA on 22, outperforming strong closed-source models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe. Qwen3-Omni adopts a Thinker-Talker MoE architecture that unifies perception and generation across text, images, audio, and video, yielding fluent text and natural real-time speech. It supports text interaction in 119 languages, speech understanding in 19 languages, and speech generation in 10 languages. To reduce first-packet latency in streaming synthesis, Talker autoregressively predicts discrete speech codecs using a multi-codebook scheme. Leveraging the representational capacity of these codebooks, we replace computationally intensive block-wise diffusion with a lightweight causal ConvNet, enabling streaming from the first codec frame. In cold-start settings, Qwen3-Omni achieves a theoretical end-to-end first-packet latency of 234 ms. To further strengthen multimodal reasoning, we introduce a Thinking model that explicitly reasons over inputs from any modality. Since the research community currently lacks a general-purpose audio captioning model, we fine-tuned Qwen3-Omni-30B-A3B to obtain Qwen3-Omni-30B-A3B-Captioner, which produces detailed, low-hallucination captions for arbitrary audio inputs. Qwen3-Omni-30B-A3B, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner are publicly released under the Apache 2.0 license.

核心贡献 · Key contributions

• 首个在文本、图像、音频和视频上均达到 SOTA 且不逊于单模态模型的全模态模型。
  First single multimodal model achieving SOTA across text, image, audio, video without degradation vs. single-modal counterparts.
• Thinker-Talker MoE 架构统一多模态感知与生成，实现低延迟流式语音。
  Thinker-Talker MoE architecture unifying perception and generation across modalities with low-latency streaming speech.
• 多码本自回归语音生成，用轻量因果 ConvNet 替代分块扩散模型。
  Multi-codebook autoregressive speech generation with lightweight causal ConvNet replacing block-wise diffusion.
• Thinking 模型实现显式多模态推理，提升文本、视觉、音频复杂任务性能。
  Thinking model for explicit multimodal reasoning, improving complex task performance across text, vision, audio.
• Qwen3-Omni-30B-A3B-Captioner 生成详细、低幻觉的音频描述，填补研究空白。
  Qwen3-Omni-30B-A3B-Captioner for detailed, low-hallucination audio captioning, filling a research gap.
• Apache 2.0 开源发布，支持 119 种文本语言、19 种语音理解和 10 种语音生成语言。
  Open-source release under Apache 2.0, supporting 119 text languages, 19 speech understanding, 10 speech generation languages.

局限 · Limitations

• 因位置外推能力和上下文长度有限，长视频基准性能欠佳。
  Suboptimal performance on long video benchmarks due to limited positional extrapolation and context length.
• Thinking 模型在 ASR 和音乐理解等感知任务上表现较差，易产生幻觉。
  Thinking model underperforms on perception tasks like ASR and music understanding, introducing hallucinations.
• 因实验成本过高，未对所有模型规模进行完整消融研究。
  No comprehensive ablation across all model scales due to prohibitive experimental cost.
• 音频数据未显著提升语言能力，增益仅限于视觉任务。
  Audio data does not measurably improve language ability; gains from adding audio are limited to vision tasks.
• 当前模型缺少多说话人 ASR、视频 OCR 和音视频主动学习能力。
  Current model lacks multi-speaker ASR, video OCR, and audiovisual proactive learning capabilities.

论文章节 · Sections（共 20）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 架构 2 Architecture
• 2.1 概述 2.1 Overview
• 2.2 音频变换器 2.2 Audio Transformer (AuT)
• 2.3 感知模块 2.3 Perceivation
• 2.4 语音生成 2.4 Speech Generation
• 2.5 流式与并发设计 2.5 Designs for Streaming and Concurrency
• 3 预训练 3 Pretraining
• 4.1 思考器 4.1 Thinker
• 4.2 说话器 4.2 Talker
• 4.3 描述器 4.3 Captioner
• 5 评估 5 Evaluation
• 5.1 X→文本评估 5.1 Evaluation of X→\toText
• 5.2 X→语音评估 5.2 Evaluation of X→\toSpeech
• 6 跨模态非退化评估 6 Evaluating Non‑Degradation Across Modalities
• 7 结论 7 Conclusion
• 8 作者 8 Authors
• 9.1 语音与音乐理解更多评估 9.1 More Evaluation on Speech and Music Understanding
• 9.2 Qwen3-Omni-30B-A3B-描述器定性结果 9.2 Qualitative Results from Qwen3-Omni-30B-A3B-Captioner

阅读逐段中英对照全文 →