Qwen2.5-Omni Technical Report
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→本报告介绍了 Qwen2.5-Omni，一个端到端的多模态模型，能够感知文本、图像、音频和视频等多种模态，同时以流式方式生成文本和自然语音响应。为了实现多模态信息输入的流式处理，音频和视觉编码器均采用分块处理方式。为了同步视频输入与音频的时间戳，我们以交错方式顺序组织音频和视频，并提出了一种新颖的位置嵌入方法，名为 TMRoPE（时间对齐多模态 RoPE）。为了同时生成文本和语音并避免两种模态之间的干扰，我们提出了 Thinker-Talker 架构。在该架构中，Thinker 作为大语言模型负责文本生成，而 Talker 是一个双轨自回归模型，直接利用 Thinker 的隐藏表示输出音频令牌。Thinker 和 Talker 模型均设计为端到端训练和推理。为了以流式方式解码音频令牌，我们引入了滑动窗口 DiT 来限制感受野，以减少初始包延迟。Qwen2.5-Omni 与同等规模的 Qwen2.5-VL 性能相当，并优于 Qwen2-Audio。此外，Qwen2.5-Omni 在 Omni-Bench 等多模态基准测试中达到了最先进的性能。值得注意的是，Qwen2.5-Omni 在端到端语音指令遵循方面的性能与文本输入相当，MMLU 和 GSM8K 等基准测试也证实了这一点。在语音生成方面，Qwen2.5-Omni 的流式 Talker 在鲁棒性和自然度上优于大多数现有流式和非流式替代方案。
In this report, we present Qwen2.5-Omni, an end-to-end multimodal model designed to perceive diverse modalities, including text, images, audio, and video, while simultaneously generating text and natural speech responses in a streaming manner. To enable the streaming of multimodal information inputs, both audio and visual encoders utilize a block-wise processing approach. To synchronize the timestamps of video inputs with audio, we organize the audio and video sequentially in an interleaved manner and propose a novel position embedding approach, named TMRoPE(Time-aligned Multimodal RoPE). To concurrently generate text and speech while avoiding interference between the two modalities, we propose \textbf{Thinker-Talker} architecture. In this framework, Thinker functions as a large language model tasked with text generation, while Talker is a dual-track autoregressive model that directly utilizes the hidden representations from the Thinker to produce audio tokens as output. Both the Thinker and Talker models are designed to be trained and inferred in an end-to-end manner. For decoding audio tokens in a streaming manner, we introduce a sliding-window DiT that restricts the receptive field, aiming to reduce the initial package delay. Qwen2.5-Omni is comparable with the similarly sized Qwen2.5-VL and outperforms Qwen2-Audio. Furthermore, Qwen2.5-Omni achieves state-of-the-art performance on multimodal benchmarks like Omni-Bench. Notably, Qwen2.5-Omni's performance in end-to-end speech instruction following is comparable to its capabilities with text inputs, as evidenced by benchmarks such as MMLU and GSM8K. As for speech generation, Qwen2.5-Omni's streaming Talker outperforms most existing streaming and non-streaming alternatives in robustness and naturalness.