Once-for-All: Train One Network and Specialize it for Efficient Deployment
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→我们解决了跨多种设备和资源约束（尤其是边缘设备）的高效推理这一难题。传统方法要么手动设计，要么使用神经架构搜索（NAS）为每种情况寻找专门的神经网络并从头训练，这在计算上代价高昂（产生的二氧化碳排放量相当于 5 辆汽车的终身排放），因此难以扩展。在这项工作中，我们提出通过解耦训练和搜索来训练一个一次训练（OFA）网络，该网络支持多种架构设置，从而降低成本。我们可以从 OFA 网络中快速选择一个专门的子网络，无需额外训练。为了高效训练 OFA 网络，我们还提出了一种新颖的渐进式收缩算法，这是一种通用的剪枝方法，可以在比剪枝更多的维度（深度、宽度、核大小和分辨率）上减小模型大小。它可以获得数量惊人的子网络（>10^19），这些子网络能够适应不同的硬件平台和延迟约束，同时保持与独立训练相同的准确率。在各种边缘设备上，OFA 始终优于最先进的 NAS 方法（在 ImageNet top1 准确率上比 MobileNetV3 提升高达 4.0%，或相同准确率但比 MobileNetV3 快 1.5 倍，比 EfficientNet 快 2.6 倍，以实测延迟计），同时减少了多个数量级的 GPU 时间和二氧化碳排放。特别是，OFA 在移动设置（<600M MACs）下实现了新的最先进水平 80.0%的 ImageNet top-1 准确率。OFA 是第三届低功耗计算机视觉挑战赛（LPCVC）DSP 分类赛道和第四届 LPCVC 分类赛道及检测赛道的获胜方案。代码和 50 个预训练模型（适用于多种设备和多种延迟约束）已在 https://github.com/mit-han-lab/once-for-all 开源。
We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing $CO_2$ emission as much as 5 cars' lifetime) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub-networks ($> 10^{19}$) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and $CO_2$ emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting ($<$600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and 50 pre-trained models (for many devices & many latency constraints) are released at https://github.com/mit-han-lab/once-for-all.