We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.
核心贡献 · Key contributions
识别出现代深度学习任务中随模型规模增大出现的双重下降现象。 Identifies double descent in modern deep learning tasks as model size increases.
展示了随训练时间增加，测试误差先升后降的逐轮双重下降。 Shows epoch-wise double descent where test error peaks then drops with training time.
定义有效模型复杂度（EMC）以统一模型级和逐轮双重下降。 Defines Effective Model Complexity (EMC) to unify model-wise and epoch-wise double descent.
证明在临界区间增加训练样本可能损害测试性能。 Demonstrates that increasing training samples can hurt test performance in critical regimes.
跨架构、数据集和优化器提供了广泛的经验验证。 Provides extensive empirical validation across architectures, datasets, and optimizers.
调和了经典偏差-方差权衡与现代过参数化深度学习。 Reconciles classical bias-variance trade-off with modern overparameterized deep learning.
局限 · Limitations
双重下降在标签噪声下最显著；无噪声设置中不太明显。 Double descent is most pronounced with label noise; less clear in noiseless settings.
有效模型复杂度是启发式定义的，缺乏 epsilon 的原则性选择。 Effective Model Complexity is defined heuristically with no principled choice of epsilon.
深度网络中双重下降的理论理解仍不完整。 Theoretical understanding of double descent in deep networks remains incomplete.
最优早停可能缓解双重下降，但其相互作用尚未充分探索。 Optimal early stopping may mitigate double descent, but its interaction is not fully explored.
结果是经验性的；缺乏'充分小/大'的形式化规范。 Results are empirical; formal specification of 'sufficiently smaller/larger' is lacking.