NVILA：高效前沿视觉语言模型

NVILA: Efficient Frontier Visual Language Models

韩松 Song Han · NVIDIA / MIT · 2024-12-05 · arXiv:2412.04468 ↗ · 被引 222

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

视觉语言模型（VLM）近年来在准确性方面取得了显著进展，但其效率问题却较少受到关注。本文介绍了 NVILA，一个旨在联合优化效率与准确性的开放 VLM 系列。基于 VILA，我们通过首先扩展空间和时间分辨率，然后压缩视觉令牌来改进模型架构。这种“先扩展后压缩”的方法使 NVILA 能够高效处理高分辨率图像和长视频。我们进一步进行了系统研究，从训练、微调到部署，全面提升了 NVILA 的效率。在广泛的图像和视频基准测试中，NVILA 在准确性上匹配或超越了领先的开放和专有 VLM。同时，它将训练成本降低了 1.9-5.1 倍，预填充延迟降低了 1.6-2.2 倍，解码延迟降低了 1.2-2.8 倍。我们公开了代码和模型以促进可重复性。

Visual language models (VLMs) have made significant advances in accuracy in recent years. However, their efficiency has received much less attention. This paper introduces NVILA, a family of open VLMs designed to jointly optimize efficiency and accuracy. Building on top of VILA, we improve its model architecture by first scaling up the spatial and temporal resolutions, and then compressing visual tokens. This "scale-then-compress" approach enables NVILA to efficiently process high-resolution images and long videos. We further conduct a systematic investigation that enhances NVILA's efficiency throughout its entire lifecycle, from training and fine-tuning to deployment. NVILA matches or surpasses the accuracy of leading open and proprietary VLMs across a wide range of image and video benchmarks. At the same time, it reduces training cost by 1.9-5.1x, prefilling latency by 1.6-2.2x, and decoding latency by 1.2-2.8x. We release our code and models to facilitate reproducibility.

核心贡献 · Key contributions

• 提出 NVILA，一系列通过“先扩展后压缩”范式同时优化效率和准确性的开放视觉语言模型。
  Proposes NVILA, a family of open VLMs optimizing both efficiency and accuracy via a 'scale-then-compress' paradigm.
• 引入 Dynamic-S2 实现自适应多尺度高分辨率图像处理，在文本密集型基准上准确率提升高达 30%。
  Introduces Dynamic-S2 for adaptive multi-scale high-resolution image processing, improving accuracy on text-heavy benchmarks by up to 30%.
• 开发基于 DeltaLoss 的数据集剪枝，减少 50%训练数据同时保持竞争性准确率。
  Develops DeltaLoss-based dataset pruning, reducing training data by 50% while maintaining competitive accuracy.
• 实现视觉语言模型的 FP8 混合精度训练，相比 BF16 实现高达 2 倍加速。
  Implements FP8 mixed-precision training for VLMs, achieving up to 2x speedup over BF16.
• 通过高效的时间扩展和压缩，在视频基准上取得最先进结果。
  Achieves state-of-the-art results on video benchmarks with efficient temporal scaling and compression.
• 在下游任务中展现强劲性能，包括时间定位、机器人导航和医学影像。
  Demonstrates strong performance in downstream tasks including temporal localization, robotic navigation, and medical imaging.

局限 · Limitations

• NVILA 的准确率提升主要在基准上验证，实际部署可能面临额外挑战。
  NVILA's accuracy gains are primarily demonstrated on benchmarks; real-world deployment may face additional challenges.
• “先扩展后压缩”方法可能不适用于所有视觉语言模型架构或任务。
  The 'scale-then-compress' approach may not generalize to all VLM architectures or tasks.
• 基于 DeltaLoss 的数据集剪枝依赖参考模型，增加计算开销。
  Dataset pruning via DeltaLoss relies on a reference model, adding computational overhead.
• FP8 训练需要特定硬件（NVIDIA H100/B200），限制可访问性。
  FP8 training requires specific hardware (NVIDIA H100/B200), limiting accessibility.
• 研究未探索模型规模扩展至 150 亿参数以上的影响。
  The study does not explore the impact of scaling model size beyond 15B parameters.

论文章节 · Sections（共 16）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 方法 2 Approach
• 2.1 高效模型架构 2.1 Efficient Model Architecture
• 2.2 高效训练 2.2 Efficient Training
• 2.3 高效微调 2.3 Efficient Fine-Tuning
• 2.4 高效部署 2.4 Efficient Deployment
• 3.1 实现细节 3.1 Implementation Details
• 3.2 精度结果 3.2 Accuracy Results
• 3.3 效率结果 3.3 Efficiency Results
• 4.1 时间定位 4.1 Temporal Localization
• 4.2 机器人导航 4.2 Robotic Navigation
• 4.3 医学多模态 VILA-M3 4.3 Medical Multi-Modal VILA-M3
• 5.1 视觉语言模型 5.1 Visual Language Models
• 5.2 效率 5.2 Efficiency
• 6 结论 6 Conclusion

阅读逐段中英对照全文 →