We present SoundStream, a novel neural audio codec that can efficiently compress speech, music and general audio at bitrates normally targeted by speech-tailored codecs. SoundStream relies on a model architecture composed by a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. Training leverages recent advances in text-to-speech and speech enhancement, which combine adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized embeddings. By training with structured dropout applied to quantizer layers, a single model can operate across variable bitrates from 3kbps to 18kbps, with a negligible quality loss when compared with models trained at fixed bitrates. In addition, the model is amenable to a low latency implementation, which supports streamable inference and runs in real time on a smartphone CPU. In subjective evaluations using audio at 24kHz sampling rate, SoundStream at 3kbps outperforms Opus at 12kbps and approaches EVS at 9.6kbps. Moreover, we are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency, which we demonstrate through background noise suppression for speech.
核心贡献 · Key contributions
提出 SoundStream，一种端到端训练的神经音频编解码器，结合对抗损失和重建损失以实现卓越音频质量。 Proposes SoundStream, a neural audio codec trained end-to-end with adversarial and reconstruction losses for superior audio quality.
引入带量化器 dropout 的残差向量量化器，实现 3 至 18 kbps 的比特率可扩展性。 Introduces a residual vector quantizer with quantizer dropout for bitrate scalability from 3 to 18 kbps.
证明学习编码器相比固定梅尔频谱特征显著提升编码效率。 Demonstrates that learning the encoder significantly improves coding efficiency over fixed mel-spectrogram features.
在主观评估中，SoundStream 在广泛比特率范围内优于 Opus 和 EVS。 Shows SoundStream outperforms Opus and EVS across a wide range of bitrates in subjective evaluations.
支持低延迟流式推理，在智能手机 CPU 上实时运行。 Supports streamable inference with low latency, running in real-time on a smartphone CPU.
实现无额外延迟的联合压缩与增强，通过背景噪声抑制演示。 Enables joint compression and enhancement without additional latency, demonstrated via background noise suppression.
局限 · Limitations
评估限于 24 kHz 采样率，未探索更高采样率下的性能。 Evaluation limited to 24 kHz sampling rate; performance at higher rates not explored.
主观测试采用众包 MUSHRA 方法，可能引入变异性。 Subjective tests use crowd-sourced MUSHRA methodology, which may introduce variability.
联合压缩与增强仅针对背景噪声抑制演示，未测试其他增强类型。 Joint compression and enhancement only demonstrated for background noise suppression; other enhancements not tested.
模型复杂度和内存需求可能限制在极低资源设备上的部署。 Model complexity and memory requirements may limit deployment on very low-resource devices.
比特率可扩展性依赖量化器 dropout，未评估极端比特率（如低于 3 kbps）下的性能。 Bitrate scalability relies on quantizer dropout; performance at extreme bitrates (e.g., below 3 kbps) not evaluated.
论文章节 · Sections（共 20）
摘要Abstract
一、引言I Introduction
二、相关工作II Related work
三、模型III Model
三-A 编码器架构III-A Encoder architecture
三-B 解码器架构III-B Decoder architecture
三-C 残差矢量量化器III-C Residual Vector Quantizer:
三-D 判别器架构III-D Discriminator architecture
三-E 训练目标III-E Training objective
三-F 联合压缩与增强III-F Joint compression and enhancement
四-A 数据集IV-A Datasets
四-B 评估指标IV-B Evaluation metrics
四-C 基线方法IV-C Baselines
五-B 客观质量指标V-B Objective quality metrics
五-C 比特率可扩展性V-C Bitrate scalability
五-D 消融研究V-D Ablation studies
五-E 联合压缩与增强V-E Joint compression and enhancement
五-F 联合与分离压缩增强对比V-F Joint vs. disjoint compression and enhancement