The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
核心贡献 · Key contributions
提出 Transformer，一种完全基于注意力机制的新架构，消除了循环和卷积。 Proposes Transformer, a novel architecture based solely on attention mechanisms, eliminating recurrence and convolutions.
在 WMT 2014 英德和英法翻译任务上分别取得 28.4 和 41.8 BLEU 的最新结果。 Achieves state-of-the-art BLEU scores of 28.4 on WMT 2014 English-German and 41.8 on English-French translation.
由于并行化，训练速度显著提升，在 8 个 GPU 上仅需 3.5 天。 Demonstrates significantly faster training due to parallelization, reducing training time to 3.5 days on 8 GPUs.
引入多头注意力机制，从不同表示子空间联合关注信息。 Introduces multi-head attention to jointly attend to information from different representation subspaces.
在英语成分句法分析等任务上表现出强泛化能力，无论数据量大小。 Shows strong generalization to other tasks like English constituency parsing with both large and limited data.
局限 · Limitations
自注意力在序列长度上的二次复杂度限制了其扩展到极长序列。 Self-attention's quadratic complexity in sequence length limits scalability to very long sequences.
模型缺乏显式位置编码，依赖附加的正弦信号，可能无法捕捉所有位置细节。 The model lacks explicit positional encoding, relying on added sinusoidal signals which may not capture all positional nuances.
训练需要仔细调整超参数（如学习率计划、dropout）才能达到最优性能。 Training requires careful hyperparameter tuning (e.g., learning rate schedule, dropout) to achieve optimal performance.
尽管有注意力可视化，架构的可解释性仍有限，深入理解学习模式仍有挑战。 The architecture's interpretability is limited despite attention visualization; deeper understanding of learned patterns remains challenging.