We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.
核心贡献 · Key contributions
提出了一种实时、高保真的神经音频编解码器，采用流式编码器-解码器和量化潜在空间。 Introduced a real-time, high-fidelity neural audio codec with streaming encoder-decoder and quantized latent space.
使用单一多尺度频谱图对抗器简化训练，减少伪影并生成高质量样本。 Simplified training with a single multiscale spectrogram adversary that reduces artifacts and produces high-quality samples.
提出了一种新颖的损失平衡器机制，通过将损失权重与其典型尺度解耦来稳定训练。 Proposed a novel loss balancer mechanism that stabilizes training by decoupling loss weights from their typical scale.
展示了轻量级 Transformer 模型可进一步压缩表示高达 40%，同时保持快于实时的速度。 Demonstrated lightweight Transformer models can further compress representations by up to 40% while staying faster than real time.
在多个带宽和音频领域（包括语音和音乐）取得了最先进的主观和客观结果。 Achieved state-of-the-art subjective and objective results across multiple bandwidths and audio domains, including speech and music.
对各种设计选择提供了广泛的消融研究和主观评估（MUSHRA）。 Provided extensive ablation studies and subjective evaluations (MUSHRA) for various design choices.
局限 · Limitations
对于训练集中未充分代表的音频领域，模型性能可能下降。 The model's performance may degrade for audio domains not well represented in the training set.
流式设置相比非流式版本会引入轻微的质量下降。 The streaming setup introduces a small degradation in quality compared to the non-streamable version.
熵编码增加了初始延迟，可能不适合低延迟应用。 The entropy coding increases initial latency, which may not be suitable for low-latency applications.
48 kHz 模型在单 CPU 核心上慢于实时，限制了其实时流式应用。 The 48 kHz model is slower than real time on a single CPU core, limiting its use for real-time streaming.
Transformer 语言模型的压缩增益随带宽变化，在较高比特率下效果较差。 The Transformer language model's compression gain varies with bandwidth, being less effective at higher bitrates.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 相关工作2 Related Work
3 模型3 Model
3.1 编码器与解码器架构3.1 Encoder & Decoder Architecture
3.2 残差向量量化3.2 Residual Vector Quantization
3.3 语言建模与熵编码3.3 Language Modeling and Entropy Coding