Andrej Karpathy 的博客：https://karpathy.github.io/2019/04/25/recipe/ 几周前，我发了一条关于“最常见的神经网络错误”的推文，列举了一些与训练神经网络相关的常见陷阱。这条推文获得的关注远超我的预期（包括一次网络研讨会）。显然，很多人都亲身经历过“卷积层是这样工作的”和“我们的卷积网络达到了最先进的水平”之间的巨大差距。因此，我想掸去博客上的灰尘，把我的推文扩展成这个主题应得的长文。然而，与其继续列举更多常见错误或详细阐述它们，我更想深入探讨如何从根本上避免这些错误（或快速修复它们）。诀窍在于遵循某个特定流程，据我所知，这个流程并没有被经常记录下来。让我们从两个重要的观察开始，正是它们引出了这个秘诀。
Andrej Karpathy's blog: https://karpathy.github.io/2019/04/25/recipe/ A few weeks ago, I posted a tweet on “the most common neural net mistakes”, listing a few common gotchas related to training neural nets. The tweet got quite a bit more engagement than I anticipated (including a webinar). Clearly, a lot of people have personally encountered the large gap between “here is how a convolutional layer works” and “our convnet achieves state of the art results”. So I thought it could be fun to brush off my dusty blog to expand my tweet to the long form that this topic deserves. However, instead of going into an enumeration of more common errors or fleshing them out, I wanted to dig a bit deeper and talk about how one can avoid making these errors altogether (or fix them very fast). The trick to doing so is to follow a certain process, which as far as I can tell is not very often documented. Let’s start with two important observations that motivate it.
核心贡献 · Key contributions
提出一套系统化、迭代式的神经网络训练流程，强调调试与验证，而非临时调参。 Proposes a systematic, iterative recipe for training neural networks, emphasizing debugging and verification rather than ad-hoc tuning.
强调在写任何模型代码前彻底检查数据，将数据集视为最终的事实来源。 Stresses thorough data inspection before writing any model code, treating the dataset as the ultimate source of truth.
建议构建可靠的训练与评估框架，使用固定随机种子、简单基线和损失/准确率合理性检查。 Recommends building a reliable training and evaluation skeleton with fixed seeds, simple baselines, and loss/accuracy sanity checks.
主张先过拟合单个批次来验证流程正确性，再增加模型复杂度。 Advocates overfitting a single batch to verify pipeline correctness before scaling up model complexity.
引入两阶段开发策略：先让大模型过拟合，再通过正则化提升验证性能。 Introduces a two-stage development strategy: first make a large model overfit, then regularize to improve validation performance.
分享实用技巧，如早期基线用 Adam、随机搜索、集成模型以及让模型训练更久。 Shares practical tips such as Adam for early baselines, random search, ensembles, and letting models train for longer.
局限 · Limitations
基于个人经验而非系统性实证研究，结论可能无法推广到所有领域。 Based on personal anecdotal experience rather than systematic empirical study; findings may not generalize across all domains.
主要聚焦小规模 CNN 的监督计算机视觉，对现代预训练、NLP 或强化学习的指导有限。 Focuses mainly on supervised computer vision with small-scale CNNs, offering limited guidance for modern pre-training, NLP, or RL.
诸如“别当英雄”和“Adam 是安全的”等建议，在当今深度学习实践中可能过于简单或过时。 Advice such as “don't be a hero” and “Adam is safe” may be too simplistic or outdated in current deep learning practice.
没有理论保证，成功高度依赖人的耐心、视觉直觉和手动检查。 Provides no theoretical guarantees; success relies heavily on human patience, visual intuition, and manual inspection.
假设单模型训练流程，未涉及分布式训练、大规模算力或复杂智能体系统。 Assumes a single-model training pipeline and does not address distributed training, large-scale compute, or complex agentic systems.
论文章节 · Sections（共 12）
概述Overview
神经网络训练是一种泄漏的抽象1) Neural net training is a leaky abstraction
征服世界conquer world here
2) 神经网络训练悄然失败2) Neural net training fails silently
配方The recipe
1. 与数据融为一体1. Become one with the data
2. 搭建端到端的训练/评估骨架并获取简单基线2. Set up the end-to-end training/evaluation skeleton + get dumb baselines