While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech, NLP or computer vision. The core idea is to predict latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which are local in nature, data2vec predicts contextualized latent representations that contain information from the entire input. Experiments on the major benchmarks of speech recognition, image classification, and natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.
核心贡献 · Key contributions
提出 data2vec，一个统一的自我监督学习框架，适用于语音、视觉和语言，使用相同的学习目标。 Proposes data2vec, a unified self-supervised learning framework for speech, vision, and language using the same objective.
通过标准 Transformer 的自蒸馏，从掩码输入预测上下文化的潜在表示。 Predicts contextualized latent representations from masked input via self-distillation with a standard Transformer.
在 ImageNet-1K、Librispeech 和 GLUE 基准上取得最先进或具有竞争力的结果。 Achieves state-of-the-art or competitive results on ImageNet-1K, Librispeech, and GLUE benchmarks.
使用教师网络的多层目标平均，优于 BYOL 等单层目标方法。 Uses multi-layer target averaging from teacher network, improving over single-layer targets like BYOL.
首个预测连续潜在目标而非离散 token 的 NLP 模型，实现开放词汇学习。 First NLP model to predict continuous latent targets instead of discrete tokens, enabling open-vocabulary learning.
证明来自完整输入的上下文化目标在自我监督学习中优于局部目标。 Demonstrates that contextualized targets from full input outperform local targets in self-supervised learning.
局限 · Limitations
仍使用模态特定的特征编码器和掩码策略，未完全统一。 Still uses modality-specific feature encoders and masking strategies, not fully unified.
需要仔细调整学习率和动量等超参数以避免表示崩溃。 Requires careful tuning of hyperparameters like learning rate and momentum to avoid representation collapse.
语音识别性能依赖大量预训练数据（960 小时），可能无法泛化到低资源语言。 Performance on speech recognition relies on large pre-training data (960 hours) and may not generalize to low-resource languages.
未探索多模态联合训练；仅单独学习每种模态的表示。 Does not explore multi-modal joint training; only learns representations for each modality separately.
上下文化目标需要完整的教师前向传播，相比局部目标增加了计算成本。 Contextualized targets require full teacher forward pass, increasing computational cost compared to local targets.