Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning

Yuxin Tang, Zhimin Ding, Dimitrije Jankov, Binhang Yuan, Daniel Bourgeois, Chris Jermaine

Introduction

In the current machine learning landscape, a significant "data tax" exists between where data resides and where training occurs. While the vast majority of enterprise data is stored in relational databases (RDBMS) using SQL, modern machine learning frameworks like PyTorch and TensorFlow operate in specialized environments. This separation necessitates the movement of terabytes of data across systems, a process that is slow, expensive, and complex.

Furthermore, specialized ML systems often encounter a "memory wall." When processing massive graphs or matrices that exceed GPU or system RAM, these frameworks frequently suffer from Out of Memory (OOM) crashes. This paper introduces a framework that brings automatic differentiation (auto-diff) directly into relational algebra, allowing large-scale models to be trained entirely within a database engine. By treating database operations as differentiable functions, the authors enable models to scale to billion-edge datasets that were previously unreachable for standard ML stacks.

The Problem: The Gap Between SQL and Auto-Diff

While some modern databases have integrated basic ML capabilities, they generally lack the engine that powers backpropagation: automatic differentiation. Without auto-diff, researchers must manually derive and code gradients for every new model architecture, a task that is error-prone and limits experimentation.

Additionally, specialized ML systems are often brittle regarding memory management. In contrast, relational engines are built to handle datasets larger than memory by utilizing sophisticated caching and "spilling" to disk. The authors argue that by embedding auto-diff into the database, we can leverage decades of database optimization research to solve the scalability challenges of modern AI.

Methodology: Functional Relational Algebra

The authors propose a three-pillared approach to integrate differentiation into the relational model:

  1. Functional Relational Algebra (RA): The system redefines standard relational operations—such as TableScan, Selection, Aggregation, and Join—as higher-order functions. This transformation allows database queries to be treated as mathematical functions suitable for differentiation.
  2. Relation-Jacobian Products (RJP): Standard deep learning relies on Vector-Jacobian Products (VJP) to propagate gradients. The authors introduce the Relation-Jacobian Product (RJP), which adapts this logic to operate on relational tables. They provide specific algorithmic "recipes" to compute the RJP for every major SQL operation.
  3. The RAAutoDiff Algorithm: This is a reverse-mode differentiation algorithm that takes a relational query (the forward pass) and automatically generates a corresponding relational query (the backward pass) to compute gradients.

Because the output of RAAutoDiff is itself a relational query, the database’s built-in query optimizer can automatically determine the most efficient way to parallelize the computation, distribute it across a cluster, or manage memory usage.

Experimental Evaluation

The researchers evaluated their system, RA-GCN, against state-of-the-art specialized systems like DistDGL and AliGraph across several massive graph datasets:

  • ogbn-papers100M: 0.1 billion nodes and 1.6 billion edges.
  • Friendster: A social network graph with 65.6 million nodes and 3.6 billion edges.
  • Freebase: A knowledge graph with 1.9 billion triples.

Key Results and Insights

  • The Optimizer as an ML Engineer: A core insight of this work is that database optimizers are naturally proficient at "tensor parallelism." When joining large tables, the optimizer automatically chooses between broadcasting or co-partitioning data based on hardware constraints—tasks that typically require manual scripting by ML engineers.
  • Graceful Scaling and the Friendster Breakthrough: RA-GCN was the only system capable of performing full-graph training on the 3.6-billion-edge Friendster graph. All other specialized systems failed with OOM errors, even when provided with 16 high-memory nodes.
  • Performance: On the ogbn-papers100M dataset, RA-GCN outperformed specialized rivals, completing training epochs in approximately 36 seconds, compared to 40–56 seconds for DistDGL and AliGraph.
  • Hardware Efficiency: RA-GCN demonstrated the ability to perform full-graph training on a single machine in scenarios where other systems required eight or more machines just to attempt mini-batch training.

Limitations and Future Work

While the framework proves the viability of in-database auto-diff, several areas for development remain. The current implementation is a Python-based tool that generates SQL; deeper integration into a database's internal kernel would likely provide even greater performance gains. Additionally, while the paper demonstrates success with Graph Convolutional Networks (GCNs) and Matrix Factorization, further research is needed to adapt the framework for the complexities of Transformer architectures and Large Language Models (LLMs). Finally, extending the framework to leverage GPU-accelerated databases remains a promising future direction to bridge the gap between CPU-based relational processing and GPU-heavy ML workloads.

Conclusion

By bringing auto-diff to the database, this research demonstrates that relational engines can serve as more than just data repositories; they can be powerful, highly scalable training environments. This approach eliminates the "data tax," prevents OOM failures through robust memory management, and leverages existing query optimization technology to automate complex distributed training logic.

Summary updated

BibTeX
@inproceedings{tang2023auto,
  author = {Yuxin Tang and Zhimin Ding and Dimitrije Jankov and Binhang Yuan and Daniel Bourgeois and Chris Jermaine},
  title = {{Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=CRmRAuwfd4},
}
Figure