We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
核心贡献 · Key contributions
提出 AudioLM 框架，通过分层结合语义和声学标记，实现高质量且长期一致的音频生成。 Proposes AudioLM, a framework combining semantic and acoustic tokens hierarchically for high-quality audio generation with long-term consistency.
证明 w2v-BERT 的语义标记捕捉语言内容，SoundStream 的声学标记确保高质量合成。 Demonstrates that semantic tokens from w2v-BERT capture linguistic content, while acoustic tokens from SoundStream ensure high-quality synthesis.
展示 AudioLM 无需文本标注即可生成句法和语义连贯的语音延续，并保留说话人身份和韵律。 Shows AudioLM generates syntactically and semantically coherent speech continuations without text annotations, preserving speaker identity and prosody.
将 AudioLM 扩展到钢琴音乐延续，无需符号音乐表示即可生成连贯的旋律和节奏。 Extends AudioLM to piano music continuation, generating coherent melodies and rhythms without symbolic music representations.
提供分类器以 98.6%的准确率检测 AudioLM 生成的语音，降低滥用风险。 Provides a classifier that detects AudioLM-generated speech with 98.6% accuracy, mitigating misuse risks.
在零样本指标 sWUGGY 和 sBLIMP 上达到最先进的词汇和句法知识，无需文本监督。 Achieves state-of-the-art lexical and syntactic knowledge in zero-shot metrics sWUGGY and sBLIMP without text supervision.
局限 · Limitations
AudioLM 仅在英语语音和钢琴音乐上评估，未展示对其他音频类型的泛化能力。 AudioLM is evaluated only on English speech and piano music; generalization to other audio types is not shown.
该框架需要预训练的标记器（w2v-BERT、SoundStream）和多阶段 Transformer，导致高计算成本。 The framework requires pre-trained tokenizers (w2v-BERT, SoundStream) and multiple Transformer stages, leading to high computational cost.
生成的语音可能反映训练数据中的社会偏见，且可能无法保留代表性不足群体的口音/方言。 Generated speech may reflect societal biases in training data and may not preserve accent/dialect for underrepresented groups.
模型保留说话人身份延续语音的能力存在欺骗生物识别或冒充他人的风险。 The model's ability to continue speech with speaker identity preservation poses risks of spoofing biometric identification or impersonation.
人工评估显示生成的语音几乎无法与真实语音区分，尽管检测准确率高，但仍引发伦理问题。 Human evaluation shows generated speech is nearly indistinguishable from real speech, raising ethical concerns despite high detection accuracy.
论文章节 · Sections（共 21）
摘要Abstract
一、引言I Introduction
二、相关工作II Related work
三、模型III Model
三-A 组件III-A Components
三-B 离散音频表示的权衡III-B Trade-offs of discrete audio representations
三-C 语义和声学标记的分层建模III-C Hierarchical modeling of semantic and acoustic tokens
三-D 推理III-D Inference
四、实验IV Experiments
四-A 数据集IV-A Datasets
四-B 模型选择、训练与推理IV-B Model selection, training and inference
四-C 语义标记表示的信息IV-C Information represented by the semantic tokens
四-D 声学标记表示的信息IV-D Information represented by the acoustic tokens
四-E 探究 AudioLM 的语言知识IV-E Probing the linguistic knowledge of AudioLM