We tackle the task of conditional music generation. We introduce MusicGen, a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns, which eliminates the need for cascading several models, e.g., hierarchically or upsampling. Following this approach, we demonstrate how MusicGen can generate high-quality samples, both mono and stereo, while being conditioned on textual description or melodic features, allowing better controls over the generated output. We conduct extensive empirical evaluation, considering both automatic and human studies, showing the proposed approach is superior to the evaluated baselines on a standard text-to-music benchmark. Through ablation studies, we shed light over the importance of each of the components comprising MusicGen. Music samples, code, and models are available at https://github.com/facebookresearch/audiocraft
核心贡献 · Key contributions
提出 MusicGen，一种单阶段 Transformer 语言模型，结合高效词元交错模式，用于高质量音乐生成。 Introduces MusicGen, a single-stage transformer LM with efficient token interleaving patterns for high-quality music generation.
在单一模型中实现文本与旋律条件控制，增强对生成音乐的控制能力。 Demonstrates text and melody conditioning in a single model, enabling better control over generated music.
提出通过码本交错模式建模多并行声学词元流的通用框架。 Proposes a general framework for modeling multiple parallel streams of acoustic tokens via codebook interleaving patterns.
利用码本模式将生成扩展到立体声音频，且不增加额外计算成本。 Extends generation to stereo audio without extra computational cost using codebook patterns.
在 MusicCaps 基准上的人类评估中，性能优于基线方法。 Achieves superior performance over baselines on MusicCaps benchmark in human evaluation.
对码本模式、模型规模和条件方法进行了广泛的消融研究。 Provides extensive ablation studies on codebook patterns, model sizes, and conditioning methods.
局限 · Limitations
缺乏对生成结果与条件一致性的精细控制，主要依赖无分类器引导。 Lacks fine-grained control over generation adherence to conditioning, relying mostly on classifier-free guidance.
旋律条件使用无监督色度图，可能无法捕捉复杂的和声结构。 Melody conditioning uses unsupervised chromagram, which may not capture complex harmonic structures.
训练数据仅限于授权音乐，可能存在多样性不足，例如西方音乐占比过高。 Training data is limited to licensed music with potential lack of diversity, e.g., overrepresentation of Western music.
模型可能记忆训练样本，引发原创性和版权方面的担忧。 Model may memorize training examples, raising concerns about originality and copyright.
在 MusicCaps 上的评估可能因数据集中的噪声样本而产生偏差。 Evaluation on MusicCaps may be biased due to noisy samples in the dataset.
论文章节 · Sections（共 18）
摘要Abstract
1 引言1 Introduction
2 方法2 Method
2.1 音频分词化2.1 Audio tokenization
2.2 码本交错模式（见图 1）2.2 Codebook interleaving patterns (see Figure 1)
2.3 模型条件化2.3 Model conditioning
2.4 模型架构2.4 Model architecture
3.1 模型与超参数3.1 Models and hyperparameters
3.2 数据集3.2 Datasets
3.3 评估3.3 Evaluation
4 结果4 Results
4.1 与基线比较4.1 Comparison with the baselines
4.2 旋律评估4.2 Melody evaluation
4.3 立体声生成微调4.3 Fine-tuning for stereophonic generation