The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC.
核心贡献 · Key contributions
提出 Fire-Flyer AI-HPC，一种用于深度学习的成本效益型软硬件协同设计。 Proposes Fire-Flyer AI-HPC, a cost-effective hardware-software co-design for deep learning.
部署 10,000 块 PCIe A100 GPU，以一半成本和 40%能耗实现 DGX-A100 80%的性能。 Deploys 10,000 PCIe A100 GPUs achieving 80% DGX-A100 performance at half cost and 40% less energy.
开发 HFReduce 实现高效全规约，在 PCIe 架构上优于 NCCL。 Develops HFReduce for efficient allreduce, outperforming NCCL on PCIe architecture.
实现 HaiScale，通过计算与通信重叠实现可扩展的大语言模型训练。 Implements HaiScale with overlapping computation and communication for scalable LLM training.
设计 3FS 分布式文件系统，采用请求发送控制机制缓解 incast 拥塞。 Designs 3FS distributed file system with request-to-send control to mitigate incast congestion.
集成 HAI 平台实现任务调度、故障处理和灾难恢复，GPU 利用率达 99%。 Integrates HAI Platform for task scheduling, fault handling, and disaster recovery, achieving 99% GPU utilization.
局限 · Limitations
PCIe A100 与 SXM A100 之间存在性能差距，限制了峰值吞吐量。 Performance gap between PCIe A100 and SXM A100 remains, limiting peak throughput.
HFReduce 性能受限于 AMD EPYC CPU 的 PCIe 主机桥带宽。 HFReduce performance bottlenecked by AMD EPYC CPU PCIe host bridge bandwidth.
为 LLM 后期添加 NVLink 桥，但引入了额外故障点。 NVLink bridge added later for LLMs, but introduces additional failure points.
因参数调优困难，网络拥塞控制禁用了 DCQCN。 Network congestion control disabled DCQCN due to parameter tuning difficulties.
双区网络限制跨区任务同时只能运行一个，降低了灵活性。 Two-zone network limits cross-zone tasks to one at a time, reducing flexibility.
论文章节 · Sections（共 32）
摘要Abstract
一、引言I Introduction
架构中的关键技术主题Key Technical Topics in our Architecture
二-A 深度学习的发展II-A Evolution of Deep Learning
二-B 模型训练的挑战与解决方案II-B Challenges and Solutions in Models Training
二-C 当代 HPC 与 AI 集群II-C HPC and AI Clusters of This Era
二-D AI 基础设施的挑战II-D Challenges in AI Infrastructure
三、Fire-Flyer 2：我们的深度学习和早期 LLM 训练方法III Fire-Flyer 2: Our Approach For Deep Learning and Early LLM Training