Recent years have witnessed a trend that large language model (LLM) based text-to-speech (TTS) emerges into the mainstream due to their high naturalness and zero-shot capacity. In this paradigm, speech signals are discretized into token sequences, which are modeled by an LLM with text as prompts and reconstructed by a token-based vocoder to waveforms. Obviously, speech tokens play a critical role in LLM-based TTS models. Current speech tokens are learned in an unsupervised manner, which lacks explicit semantic information and alignment to the text. In this paper, we propose to represent speech with supervised semantic tokens, which are derived from a multilingual speech recognition model by inserting vector quantization into the encoder. Based on the tokens, we further propose a scalable zero-shot TTS synthesizer, CosyVoice, which consists of an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis. Experimental results show that supervised semantic tokens significantly outperform existing unsupervised tokens in terms of content consistency and speaker similarity for zero-shot voice cloning. Moreover, we find that utilizing large-scale data further improves the synthesis performance, indicating the scalable capacity of CosyVoice. To the best of our knowledge, this is the first attempt to involve supervised speech tokens into TTS models.
核心贡献 · Key contributions
首次将监督语义令牌集成到 TTS 模型中，提升了内容一致性和说话人相似度。 First integration of supervised semantic tokens into TTS models, enhancing content consistency and speaker similarity.
提出 CosyVoice，一种可扩展的零样本 TTS 合成器，结合了用于文本到令牌生成的 LLM 和用于令牌到语音合成的条件流匹配模型。 Proposes CosyVoice, a scalable zero-shot TTS synthesizer combining an LLM for text-to-token generation and a conditional flow matching model for token-to-speech synthesis.
通过直接的文本到语音路径，消除了对音素器和强制对齐器的需求。 Eliminates need for phonemizers and forced aligners by using a direct text-to-speech pathway.
将 x-vector 引入 LLM，将语音建模分离为语义、说话人和韵律组件。 Incorporates x-vector into LLM to separate modeling of speech into semantic, speaker, and prosody components.
展示了可扩展性：大规模数据和模型大小显著提升了合成性能。 Demonstrates scalability: large-scale data and model size significantly improve synthesis performance.
在零样本语音克隆和跨语言场景中实现了人类级别的生成质量。 Achieves human parity generation quality in zero-shot voice cloning and cross-lingual scenarios.
局限 · Limitations
评估仅限于英语和中文；其他语言的多语言性能尚未充分探索。 Evaluation limited to English and Chinese languages; multilingual performance on other languages not fully explored.
依赖预训练的 ASR 模型进行监督令牌提取可能引入偏差。 Dependence on a pre-trained ASR model for supervised token extraction may introduce biases.
单一码本包含 4096 个码字；码本大小和数量对性能的影响尚未研究。 Single codebook with 4096 codes; impact of codebook size and number on performance not investigated.
零样本语音克隆需要参考语音样本；对于极短或嘈杂的样本，性能可能下降。 Zero-shot voice cloning requires a reference speech sample; performance may degrade with very short or noisy samples.
指令微调数据仅限于特定情感和副语言特征；更广泛的可控性尚未测试。 Instruct fine-tuning data limited to specific emotions and paralinguistic features; broader controllability not tested.
论文章节 · Sections（共 17）
摘要Abstract
1 引言1 Introduction
2 CosyVoice：基于监督语义标记的可扩展 TTS 模型2 CosyVoice: A Scalable TTS model using Supervised Semantic Tokens
2.1 用于语音的监督语义标记2.1 Supervised Semantic Tokens for Speech