RDBench: ML Benchmark for Relational Databases

Tao Feng

TL;DR

Machine learning on relational databases (RDBs) has been hampered by a lack of standardized benchmarks. This paper introduces RDBench, a comprehensive benchmark suite with diverse RDB datasets, unified task definitions, and multiple data representations (tabular, homogeneous, and heterogeneous graphs) to facilitate reproducible ML research in this domain.

What Problem Does This Paper Address?

While machine learning has seen tremendous progress across various domains like computer vision and natural language processing, its application to relational databases has lagged due to the absence of well-established benchmarks. This lack of standardization makes it difficult to compare different ML methods, hinders reproducible research, and slows down advancements in applying ML to the vast amount of data stored in RDBs. Existing work often focuses on specific aspects like database management systems or real-time feature extraction, overlooking the growing need for robust ML benchmarks for RDBs.

Methods

The core contribution of RDBench is a standardized framework for evaluating ML models on relational databases. Key aspects include:

  • Unified Task Definition: All tasks are framed as column value prediction problems, ensuring comparability across different data formats and models.
  • Multiple Data Representations: RDBench provides three ways to represent RDB data:
  • Tabular Data: Directly using tables, including single tables and joined tables (one-hop neighborhood).
  • Homogeneous Graphs: Transforming the RDB into a single-type graph.
  • Heterogeneous Graphs: Transforming the RDB into a graph with different node and edge types, preserving more structural information.
  • Hierarchical Datasets: Datasets are organized into three levels of complexity based on the number of tables (3-5, 6-10, and 10+ tables), allowing for a progressive evaluation of model capabilities.
  • DBGym Toolkit: A user-friendly API built on popular ML frameworks (like PyTorch Geometric) to simplify the adoption and benchmarking process.
  • Robust Evaluation: Results are averaged over multiple runs and target columns to ensure statistical robustness.

Datasets

RDBench introduces a suite of datasets categorized into three levels of complexity:

  • RDBench-L0 (Graph Datasets as RDBs): Two datasets (rdb0-arxiv, rdb0-molhiv) derived from existing graph ML benchmarks (OGB) to validate the graph-to-RDB transformation.
  • RDBench-L1 (Relational Databases with 3-5 Tables): Four datasets (rdb1-ather, rdb1-rscore, rdb1-accdt, rdb1-seznam) covering diverse domains like science, economics, and society.
  • RDBench-L2 (Relational Databases with 6-10 Tables): Four datasets (rdb2-hbv, rdb2-ncaa, rdb2-bank, rdb2-imdb) from domains such as health, sports, finance, and entertainment.
  • RDBench-L3 (Relational Databases with 10+ Tables): Three datasets (rdb3-toxic, rdb3-lahman, rdb3-govern) dealing with toxicology, baseball statistics, and government elections.

Key Insights and Ideas

  • Structural Information is Crucial: As the number of tables increases, graph-based methods (especially heterogeneous ones) begin to outperform traditional tabular methods like XGBoost, highlighting the growing importance of capturing complex inter-table relationships.
  • Graph Representations Unlock Potential: Transforming RDBs into graph structures allows ML models, particularly GNNs, to leverage relational information more effectively, leading to improved performance on tasks that involve complex relationships.
  • Tabular Models Still Competitive (for simpler RDBs): For RDBs with fewer tables, well-established tabular models like XGBoost remain highly competitive and sometimes superior, especially in regression tasks.
  • Heterogeneous Graphs Offer an Edge: Methods on heterogeneous graphs often outperform those on homogeneous graphs, suggesting that preserving the distinct types of nodes and edges in RDBs is beneficial.

Key Results

  • RDBench-L0: GCN and GraphSage on the RDB representation achieved results comparable to their performance on the original graph datasets, validating the transformation process.
  • RDBench-L1 & L2: XGBoost generally performed very well, often outperforming other models, especially in regression tasks. However, graph neural networks like GraphSage and HGCN showed competitive performance in classification tasks. Join XGBoost demonstrated significant gains by incorporating one-hop neighborhood information.
  • RDBench-L3: With an increasing number of tables, XGBoost's dominance waned. Graph methods, particularly HGCN and HGT, started to outperform tabular methods in several tasks, indicating the critical role of relationships in larger RDBs.
  • General Trend: In regression tasks, tree-based models like XGBoost consistently outperformed neural network-based models across most levels. In classification tasks, GNNs showed comparable performance to tabular methods, especially as relational complexity increased.

Limitations and Future Work

  • Target Leakage: The authors acknowledge that some tasks might have exhibited target leakage, where the target column was implicitly expressed by other columns, leading to unrealistically high performance. While most such tasks were removed, a few were retained for assessment.
  • Model Applicability: For some tasks, especially in RDBench-L2 and L3, certain models were not applicable when only the target table was used, as indicated by "N/A" in the results tables.
  • Future Work: The paper suggests that researchers can use RDBench to deploy models, obtain comparable results, and further enhance collaboration across different ML subdomains. The benchmark is designed to facilitate the development of new ML methods for relational databases.

Summary updated

Abstract

Benefiting from high-quality datasets and standardized evaluation metrics, machine learning (ML) has achieved sustained progress and widespread applications. However, while applying machine learning to relational databases, the absence of a well-established benchmark remains a significant obstacle to the development of ML. To address this issue, we introduce \textit{ML Benchmark For Relational Databases} (RDBench), a benchmark that aims to promote hierarchical, robust, and reproducible ML research on relational databases. RDBench offers hierarchical datasets of varying scales, domains, and relations. It provides three types of data: tabular data, homogeneous graphs, and heterogeneous graphs. Importantly, all data formats share the same task definition, allowing for meaningful comparisons between methods across different data formats. Reported results are averaged over the same datasets and tasks (classification or regression), further enhancing the robustness of the experimental findings. In addition to dataset construction, we conduct extensive experiments to uncover performance differences between models. To better present our proposed RDBench, we offer a user-friendly API that provides standardized formats for three types of data.
BibTeX
@misc{feng2023rdbench,
  author = {Tao Feng},
  title = {{RDBench: ML Benchmark for Relational Databases}},
  year = {2023},
  eprint = {2310.16837},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.16837},
}
Figure