TUDataset: A collection of benchmark datasets for learning with graphs

Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, Marion NeumannarXiv 2020 · OpenReview

TL;DR

The paper introduces TUDataset, a comprehensive collection of over 120 benchmark datasets for graph-based machine learning, addressing the critical need for standardized evaluation and diverse data. It also provides standardized evaluation protocols and baseline implementations for graph kernels and Graph Neural Networks (GNNs), enabling fairer comparisons and accelerating research in this rapidly growing field.

What Problem Does This Paper Address?

The field of graph representation learning, particularly supervised learning with graphs using Graph Neural Networks (GNNs) and graph kernels, has seen a surge in interest. However, a significant bottleneck hindering progress is the lack of standardized, diverse, and meaningful benchmark datasets. Researchers often evaluate new methods on small, non-diverse datasets using inconsistent experimental protocols. This makes it difficult to compare results across different studies, understand the true strengths and weaknesses of proposed methods, and identify which approaches generalize best. TUDataset aims to solve this by providing a unified, large-scale collection of datasets, along with standardized evaluation tools and baseline implementations.

Methods

The core contribution of this paper is the TUDataset collection, which comprises over 120 datasets spanning various domains like bioinformatics, computer vision, and social networks. These datasets are made accessible through Python-based data loaders compatible with popular frameworks like PyTorch Geometric and DGL.

Beyond the datasets themselves, the authors provide:

  • Standardized Evaluation Procedures: They propose a consistent framework for evaluating both graph kernels and GNNs. This includes using 10-fold cross-validation with a held-out validation set for hyperparameter tuning, and reporting results with standard deviations over multiple repetitions. For regression tasks, they outline specific protocols for handling train/validation/test splits and reporting metrics like MAE and logMAE.
  • Baseline Implementations: The paper includes implementations of common graph kernels (e.g., Weisfeiler-Lehman Subtree, Shortest-path, Graphlet, Weisfeiler-Lehman Optimal Assignment) and makes GNN architectures from PyTorch Geometric readily available for benchmarking.
  • Experimental Study: They conduct an experimental study comparing graph kernels and GNNs on a subset of TUDataset to demonstrate the utility of their framework and provide initial insights into the performance of different methods.

Datasets

The TUDataset collection includes over 120 datasets, categorized by application domain. Some notable examples mentioned include:

  • Small Molecules: MUTAG, PTC, NC1, NC1109, Tox21, PUBCHEM, ALCHEMY, QM9, ZINC. These are crucial for drug discovery and materials science.
  • Bioinformatics: DD, ENZYMES, PROTEINS. These represent macromolecules and are used for tasks like enzyme classification.
  • Computer Vision: LETTER, FINGERPRINT, CUNEIFORM signs, 3D point clouds, MSRC.
  • Social Networks: REDDIT datasets (various types), COLLAB, IMDB datasets, DEEZER_EGO_NETS, TWITCH_EGOS, GITHUB_STARGAZERS.
  • Temporal Graphs: TUMBLR, DBLP, FACEBOOK, MIT contacts, HIGHSCHOOL visitors.
  • Synthetic Datasets: SYNTHETICNEW, SYNTHIE, COLORS, TRIANGLES.

The paper emphasizes that these datasets cover a wide range of sizes and applications, and are available at www.graphlearning.io.

Key Insights and Ideas

  • Graph Kernels Remain Competitive: Despite the rise of GNNs, the experimental results show that classical graph kernels, when combined with SVMs, are still highly competitive for graph classification tasks, especially on smaller datasets. This suggests that GNNs don't always offer a clear advantage and the choice of method might depend on the specific task and dataset characteristics.
  • Scalability is a Major Differentiator: While some graph kernels perform well on small datasets, they often struggle to scale to larger ones (e.g., those requiring Gram matrix computation). GNNs, on the other hand, are generally designed with scalability in mind.
  • Specialized Architectures Matter for Regression: For complex tasks like molecular regression, generic GNN architectures might not be sufficient. Specialized architectures like Message Passing Neural Networks (MPNNs) show significant gains, highlighting the importance of incorporating domain-specific knowledge or inductive biases.
  • Standardization is Crucial: The paper strongly advocates for standardized evaluation protocols. Their framework aims to reduce the "reproducibility crisis" in machine learning by ensuring that new methods are compared fairly and consistently.

Key Results

  • Small-Scale Classification: On small-scale datasets (Table 1), the Weisfeiler-Lehman Optimal Assignment (WL-OA) kernel performed best overall.
  • Mid-Scale Classification: For mid-scale datasets (Table 2), the 1-WL kernel performed well across most datasets, but GNNs (specifically GIN-E and GIN-E-JK) showed superior performance on the GITHUB_STARGAZERS dataset.
  • Large-Scale Regression: In molecular regression tasks (Table 3), specialized architectures like MPNN significantly outperformed the generic GINE-ɛ baseline, achieving better Mean Absolute Error (MAE) and logMAE. For instance, on the ZINC dataset, MPNN achieved a mean MAE of 0.034 ± 0.001, compared to GINE-ɛ's 0.084 ± 0.004.

Limitations and Future Work

  • Dataset Coverage: While TUDataset is extensive, the authors acknowledge that future work will involve adding more datasets, particularly for large-scale molecular regression tasks with continuous node and edge features.
  • Method Comparison Depth: The experimental evaluation primarily focused on comparing a few representative graph kernels and GNN baselines. A more extensive comparison across a wider range of state-of-the-art methods would be beneficial.
  • State-of-the-Art Performance: The authors explicitly state that their intent in the experimental evaluation was not to beat the state-of-the-art, especially for physical knowledge-incorporating architectures in molecular regression. Their focus was on providing baseline scores within their standardized framework.
  • Future Work: The authors look forward to adding more datasets and encourage contributions from the community. They also plan for a more extensive comparison of kernel and neural approaches on large-scale molecular regression tasks.

Summary updated

Abstract

Recently, there has been an increasing interest in (supervised) learning with graph data, especially using graph neural networks. However, the development of meaningful benchmark datasets and standardized evaluation procedures is lagging, consequently hindering advancements in this area. To address this, we introduce the TUDataset for graph classification and regression. The collection consists of over 120 datasets of varying sizes from a wide range of applications. We provide Python-based data loaders, kernel and graph neural network baseline implementations, and evaluation tools. Here, we give an overview of the datasets, standardized evaluation procedures, and provide baseline experiments. All datasets are available at www.graphlearning.io. The experiments are fully reproducible from the code available at www.github.com/chrsmrrs/tudataset.
BibTeX
@misc{morris2020tudataset,
  author = {Christopher Morris and Nils M. Kriege and Franka Bause and Kristian Kersting and Petra Mutzel and Marion Neumann},
  title = {{TUDataset: A collection of benchmark datasets for learning with graphs}},
  year = {2020},
  eprint = {2007.08663},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.08663},
}
Figure