GOGGLE: Generative Modelling for Tabular Data by Learning Relational Structure

Tennison Liu, Zhaozhi Qian, Jeroen Berrevoets, Mihaela van der SchaarICLR 2023 · OpenReview

TL;DR

GOGGLE is a novel generative framework for tabular data that moves beyond standard "black-box" architectures by explicitly learning the relational structure (the "who-depends-on-whom") of variables. By jointly optimizing a graph of variable dependencies and a message-passing generative process, GOGGLE produces more realistic synthetic data, resists overfitting on small datasets, and provides a natural interface for incorporating domain expertise.

The Challenge of Tabular Data

Deep generative models (DGMs) have revolutionized image and text generation because those domains have inherent structures—like spatial locality in images or sequentiality in text—that architectures like CNNs and Transformers can easily exploit. Tabular data, however, is fundamentally different: the relationships between columns (e.g., "Age," "Income," "Diagnosis") are heterogeneous, sparse, and often unknown.

Most current tabular generators rely on Multilayer Perceptrons (MLPs). MLPs are "all-to-all" learners; they assume every variable can interact with every other variable. This lack of structural "inductive bias" leads to two major issues:

  1. Overfitting: On small datasets, MLPs tend to memorize specific samples rather than learning the underlying distribution.
  2. Inflexibility: There is no straightforward way to tell an MLP that specific variables are known to be related, or to force it to ignore spurious correlations.

Introducing GOGGLE

The authors introduce GOGGLE (Generative mOdellinG with Graph LEarning), which treats tabular generation as a graph-based inference problem.

The GOGGLE Framework

Instead of relying on fully connected layers, GOGGLE introduces several key innovations:

  • Learnable Relational Structure: GOGGLE learns a weighted adjacency matrix ($G_\phi$) rather than assuming all variables are connected. This matrix represents the strength of dependencies between different features in the table.
  • Message Passing Decoder: GOGGLE uses a Variational Autoencoder (VAE) backbone but replaces the standard MLP decoder with a Message Passing Neural Network (MPNN). To generate a specific column's value, the model "consults" its neighbors in the learned graph over several rounds of message passing.
  • Joint Optimization: The model learns the graph structure and the functional relationships (the generation logic) simultaneously end-to-end.
  • Sparsity and Priors: An $L_1$ penalty is applied to the adjacency matrix to encourage a sparse, efficient graph. Crucially, if a researcher has prior knowledge (e.g., a known biological pathway), they can regularize the learned graph toward that known structure.

Key Insights

  • Sparsity is a Feature, Not a Bug: By forcing the model to generate variables using only a small subset of "neighbor" variables, GOGGLE is effectively regularized. This prevents the model from memorizing the training set, leading to better generalization.
  • The "Aha" Moment on Small Data: GOGGLE’s advantage is most pronounced on small datasets. While massive datasets can eventually "teach" an MLP which correlations matter, GOGGLE’s structural bias allows it to find the signal much faster when data is scarce.
  • White-Box Generation: Because the learned graph is explicit, researchers can inspect the adjacency matrix to see what relationships the model has discovered (e.g., verifying that "Account Balance" depends on "Occupation" in a credit dataset).

Experimental Results

The researchers evaluated GOGGLE on 10 diverse real-world datasets, including standard UCI benchmarks (Adult, Breast Cancer, Credit Risk) and Bayesian Network datasets (ECOLI, MAGIC-IRRI) to test the incorporation of ground-truth structures.

Average Rankings Across Key Metrics
  • Superior Quality: As shown in the rankings above, GOGGLE consistently achieved the best (lowest) average rank across three key metrics: Quality (how realistic the data is), Detection (how hard it is for a classifier to distinguish synthetic from real data), and Utility (how well a model trained on synthetic data performs on real test data).
  • Prior Knowledge Boost: In experiments on the ECOLI dataset, providing even 50% of the true relational graph significantly improved synthetic data quality, especially in "low-data" regimes (under 500 samples).
  • Ablation Success: The authors proved that jointly learning the graph and the generator is critical; using a fixed graph based on simple correlation resulted in much poorer performance.

Limitations and Future Work

While highly effective, GOGGLE does face dimensionality constraints. The current study focused on datasets with fewer than 100 features. Because the adjacency matrix grows quadratically, scaling GOGGLE to high-dimensional fields like genomics remains a challenge. Additionally, learning the graph structure adds computational complexity compared to standard VAEs or GANs.

Future work will likely explore "latent" relational structures—learning relationships between groups of variables or in a compressed representation space—to handle higher-dimensional tabular data efficiently.

Summary updated

Abstract

Deep generative models learn highly complex and non-linear representations to generate realistic synthetic data. While they have achieved notable success in computer vision and natural language processing, similar advances have been less demonstrable in the tabular domain. This is partially because generative modelling of tabular data entails a particular set of challenges, including heterogeneous relationships, limited number of samples, and difficulties in incorporating prior knowledge. Additionally, unlike their counterparts in image and sequence domain, deep generative models for tabular data almost exclusively employ fully-connected layers, which encode weak inductive biases about relationships between inputs. Real-world data generating processes can often be represented using relational structures, which encode sparse, heterogeneous relationships between variables. In this work, we learn and exploit relational structure underlying tabular data to better model variable dependence, and as a natural means to introduce regularization on relationships and include prior knowledge. Specifically, we introduce GOGGLE, an end-to-end message passing scheme that jointly learns the relational structure and corresponding functional relationships as the basis of generating synthetic samples. Using real-world datasets, we provide empirical evidence that the proposed method is effective in generating realistic synthetic data and exploiting domain knowledge for downstream tasks.
BibTeX
@inproceedings{liu2023goggle,
  author = {Tennison Liu and Zhaozhi Qian and Jeroen Berrevoets and Mihaela van der Schaar},
  title = {{GOGGLE: Generative Modelling for Tabular Data by Learning Relational Structure}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=fPVRcJqspu},
}
Figure