图神经网络基准测试

Benchmarking Graph Neural Networks

泽维尔·布雷松 Xavier Bresson · NTU Singapore · 2020-03-02 · arXiv:2003.00982 ↗ · 被引 1212

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

在过去几年中，图神经网络（GNN）已成为分析和学习图数据的标准工具。这一新兴领域见证了大量有前景技术的广泛增长，这些技术已成功应用于计算机科学、数学、生物学、物理学和化学。但要使任何成功的领域成为主流并可靠，必须开发基准测试来量化进展。这促使我们在 2020 年 3 月发布了一个基准框架，该框架：i）包含多样化的数学和真实世界图集合，ii）在相同参数预算下实现公平模型比较以识别关键架构，iii）具有开源、易用且可复现的代码基础设施，iv）灵活支持研究人员实验新的理论想法。截至 2022 年 12 月，GitHub 仓库已获得 2000 颗星和 380 个分支，这证明了所提出的开源框架通过 GNN 社区的广泛使用而具有实用性。在本文中，我们介绍了基准测试的更新版本，简洁地展示了上述框架特征，添加了一个类似于流行 ZINC 的中等规模分子数据集 AQSOL，但具有真实世界测量的化学目标，并讨论了如何利用该框架探索新的 GNN 设计和见解。作为基准测试价值的证明，我们研究了 GNN 中的图位置编码（PE）案例，该案例随此基准测试引入，并已激发在稳健实验环境中探索更强大 PE 用于 Transformer 和 GNN 的兴趣。

In the last few years, graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. This emerging field has witnessed an extensive growth of promising techniques that have been applied with success to computer science, mathematics, biology, physics and chemistry. But for any successful field to become mainstream and reliable, benchmarks must be developed to quantify progress. This led us in March 2020 to release a benchmark framework that i) comprises of a diverse collection of mathematical and real-world graphs, ii) enables fair model comparison with the same parameter budget to identify key architectures, iii) has an open-source, easy-to-use and reproducible code infrastructure, and iv) is flexible for researchers to experiment with new theoretical ideas. As of December 2022, the GitHub repository has reached 2,000 stars and 380 forks, which demonstrates the utility of the proposed open-source framework through the wide usage by the GNN community. In this paper, we present an updated version of our benchmark with a concise presentation of the aforementioned framework characteristics, an additional medium-sized molecular dataset AQSOL, similar to the popular ZINC, but with a real-world measured chemical target, and discuss how this framework can be leveraged to explore new GNN designs and insights. As a proof of value of our benchmark, we study the case of graph positional encoding (PE) in GNNs, which was introduced with this benchmark and has since spurred interest of exploring more powerful PE for Transformers and GNNs in a robust experimental setting.

核心贡献 · Key contributions

• 发布了一个开源 GNN 基准测试框架，包含多样化数据集、公平比较协议和模块化代码。
  Released an open-source GNN benchmarking framework with diverse datasets, fair comparison protocols, and modular code.
• 新增 AQSOL，一个具有真实实验溶解度目标的中等规模分子数据集。
  Added AQSOL, a medium-sized molecular dataset with real-world experimental solubility targets.
• 通过研究图位置编码展示了框架的实用性，推动了进一步研究。
  Demonstrated the framework's utility by studying graph positional encoding, spurring further research.
• 通过固定参数预算（10 万和 50 万）实现了公平的模型比较。
  Enabled fair model comparison with fixed parameter budgets of 100k and 500k.
• 提供了 12 个数据集，涵盖来自不同领域的图级、节点级和边级任务。
  Provided 12 datasets covering graph, node, and edge-level tasks from diverse domains.
• 通过标准化的训练和评估流程促进了可重复研究。
  Facilitated reproducible research with standardized training and evaluation pipelines.

局限 · Limitations

• 中等规模数据集可能无法捕捉超大图的挑战。
  Medium-scale datasets may not capture challenges of very large graphs.
• 固定参数预算限制了对每个模型最优超参数的探索。
  Fixed parameter budgets limit exploration of optimal hyperparameters for each model.
• WL-GNN 因 O(n²)/O(n³)复杂度面临可扩展性问题。
  WL-GNNs face scalability issues due to O(n^2)/O(n^3) complexity.
• 基准测试未涵盖动态或时序图任务。
  Benchmark does not cover dynamic or temporal graph tasks.
• 结果可能无法泛化到数据集所代表领域之外。
  Results may not generalize to domains outside those represented in datasets.

论文章节 · Sections（共 30）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 GNN 基准测试框架概述 2 Overview of GNN Benchmarking Framework
• 3 如何利用基准测试探索新见解？ 3 How can the benchmark be used to explore new insights?
• 4 结论 4 Conclusion
• 致谢 Acknowledgments
• 附录 A 相关工作 Appendix A Related Work
• 附录 B 图神经网络流水线 Appendix B Graph Neural Network Pipeline
• B.1 消息传递 GCN B.1 Message-Passing GCNs
• B.2 Weisfeiler-Lehman GNN B.2 Weisfeiler-Lehman GNNs
• 附录 C 数据集与基准测试实验 Appendix C Datasets and Benchmarking Experiments
• C.1 基于 ZINC 数据集的图回归 C.1 Graph Regression with ZINC dataset
• C.2 基于 AQSOL 数据集的图回归 C.2 Graph Regression with AQSOL dataset
• C.3 基于 OGBL-COLLAB 数据集的链接预测 C.3 Link Prediction with OGBL-COLLAB dataset
• C.4 基于 WikiCS 数据集的节点分类 C.4 Node Classification with WikiCS dataset
• C.5 基于超像素（MNIST/CIFAR10）数据集的图分类 C.5 Graph Classification with Super-pixel (MNIST/CIFAR10) datasets
• C.6 基于 SBM（PATTERN/CLUSTER）数据集的节点分类 C.6 Node Classification with SBM (PATTERN/CLUSTER) datasets
• C.7 基于 TSP 数据集的边分类/链接预测 C.7 Edge Classification/Link Prediction with TSP dataset
• C.8 基于 CSL 数据集的图分类与同构测试 C.8 Graph Classification and Isomorphism Testing with CSL dataset
• C.9 基于 CYCLES 数据集的环检测 C.9 Cycle Detection with CYCLES dataset
• C.10 基于 GraphTheoryProp 数据集的多任务图属性 C.10 Multi-task graph properties with GraphTheoryProp dataset
• 附录 D 基准测试结果分析与讨论 Appendix D Analysis and Discussion of Benchmarking Results
• 附录 E 基于基准测试框架的研究 Appendix E Studies using the Benchmarking Framework
• E.1 拉普拉斯位置编码 E.1 Laplacian Positional Encodings
• E.2 用于链接预测的边表示 E.2 Edge representations for link prediction.
• 附录 F 在 TU 数据集上的实验 Appendix F Experiments on TU datasets
• 附录 G 关于图大小归一化的说明 Appendix G A Note on Graph Size Normalization
• 附录 H 基准测试设计选择的详细说明 Appendix H Elaboration on Benchmarking Design Choices
• 附录 I 硬件 Appendix I Hardware
• 附录 J 内存使用 Appendix J Memory Usage

阅读逐段中英对照全文 →