Automatic Bayesian Density Analysis

Antonio Vergari, Alejandro Molina, Robert Peharz, Zoubin Ghahramani, Kristian Kersting, Isabel Valera

Abstract

Making sense of a dataset in an automatic and unsupervised fashion is a challenging problem in statistics and AI. Classical approaches for exploratory data analysis are usually not flexible enough to deal with the uncertainty inherent to real-world data: they are often restricted to fixed latent interaction models and homogeneous likelihoods; they are sensitive to missing, corrupt and anomalous data; moreover, their expressiveness generally comes at the price of intractable inference. As a result, supervision from statisticians is usually needed to find the right model for the data. However, since domain experts are not necessarily also experts in statistics, we propose Automatic Bayesian Density Analysis (ABDA) to make exploratory data analysis accessible at large. Specifically, ABDA allows for automatic and efficient missing value estimation, statistical data type and likelihood discovery, anomaly detection and dependency structure mining, on top of providing accurate density estimation. Extensive empirical evidence shows that ABDA is a suitable tool for automatic exploratory analysis of mixed continuous and discrete tabular data.

Introduction: The Bottleneck of Data Understanding

Before building predictive models, data scientists must perform Exploratory Data Analysis (EDA). This critical phase involves identifying data types (e.g., Gaussian vs. Poisson), uncovering dependencies, handling missing values, and spotting outliers. Currently, this process is a significant bottleneck, requiring manual intervention from statisticians to select appropriate models for heterogeneous data. While automated tools exist for supervised tasks like classification, unsupervised EDA remains largely manual. Existing models like CrossCat or Mixed Sum-Product Networks (MSPNs) often struggle with scalability, lack robustness to noisy structures, or require users to pre-specify statistical distributions for every feature.

Automatic Bayesian Density Analysis (ABDA) addresses these challenges by providing a fully unsupervised framework that automates the initial steps of data science. It serves as a "one-stop-shop" for data understanding, capable of simultaneously imputing missing values, detecting anomalies, and discovering underlying statistical types without human intervention.

Overview of ABDA's capabilities: (a) raw data with missing values, (b) discovered clusters and types, (c) inferred likelihoods, (d) distribution statistics, and (e) extracted rules.

Methodology

ABDA is built upon Sum-Product Networks (SPNs), deep probabilistic models that facilitate exact and efficient inference. The framework introduces several key innovations to standard SPN architectures:

  1. Type-Augmented SPNs: Unlike standard SPNs that utilize fixed leaf distributions, ABDA employs a "dictionary" of likelihood models (e.g., Gaussian, Gamma, Poisson) at the leaves. The choice of the correct distribution is treated as a latent variable to be inferred during training.
  2. Bayesian Inference via Gibbs Sampling: While most SPNs are trained using Maximum Likelihood—which is prone to overfitting—ABDA adopts a Bayesian approach. Using Gibbs sampling for parameter learning allows the model to quantify uncertainty and remain robust even when the provided structure is overly complex for the data.
  3. Hierarchical Co-clustering: The model naturally partitions data by both rows (samples) and columns (features). This enables the discovery of context-specific dependencies, such as correlations that only exist within specific sub-populations of the dataset.
Structural representation of ABDA: (a) the plate notation graphical model, (b) the corresponding Sum-Product Network (SPN), and (c) the latent variable (LV) interpretation.

Key Insights and Empirical Results

One of the most significant findings is that the Bayesian prior on SPN weights acts as a natural regularizer. If a user provides an overly complex model structure, the Gibbs sampler effectively "shuts down" unnecessary parts of the network, resulting in a sparser and more accurate model.

ABDA's ability to discover meaningful sub-populations was demonstrated on the Wine dataset, where it successfully identified red and white wines as distinct modes and assigned appropriate likelihood models (Gaussian vs. Gamma) based on chemical properties—all without being provided with a "wine color" label.

Performance Summary

  • Density Estimation: ABDA consistently outperformed state-of-the-art competitors like ISLV and MSPNs in test log-likelihood across 12 real-world UCI datasets, particularly in inductive settings (predicting on unseen data).
  • Imputation: In missing value estimation, ABDA showed lower Normalized Root Mean Square Error (NRMSE) and higher log-likelihoods than competitors, proving more robust to heterogeneous data types.
  • Anomaly Detection: Despite being a general-purpose density estimator, ABDA performed comparably to—and often better than—specialized algorithms like One-Class SVMs and Local Outlier Factor (LOF) across 10 benchmark datasets.
  • Computational Efficiency: By exploiting the SPN structure, ABDA's Gibbs sampling is orders of magnitude faster than previous latent variable models, making it practical for real-world applications.

Conclusion and Future Work

ABDA provides a unified probabilistic backbone for multiple EDA tasks. While it automates much of the process, the choice of priors for likelihood models still benefits from sensible defaults. Future work aims to integrate ABDA into pipelines that generate human-readable reports, translating discovered probabilistic patterns into natural language for domain experts, further lowering the barrier to sophisticated data analysis.

Summary updated

Abstract

Making sense of a dataset in an automatic and unsupervised fashion is a challenging problem in statistics and AI. Classical approaches for {exploratory data analysis} are usually not flexible enough to deal with the uncertainty inherent to real-world data: they are often restricted to fixed latent interaction models and homogeneous likelihoods; they are sensitive to missing, corrupt and anomalous data; moreover, their expressiveness generally comes at the price of intractable inference. As a result, supervision from statisticians is usually needed to find the right model for the data. However, since domain experts are not necessarily also experts in statistics, we propose Automatic Bayesian Density Analysis (ABDA) to make exploratory data analysis accessible at large. Specifically, ABDA allows for automatic and efficient missing value estimation, statistical data type and likelihood discovery, anomaly detection and dependency structure mining, on top of providing accurate density estimation. Extensive empirical evidence shows that ABDA is a suitable tool for automatic exploratory analysis of mixed continuous and discrete tabular data.
BibTeX
@inproceedings{vergari2019automatic,
  author = {Antonio Vergari and Alejandro Molina and Robert Peharz and Zoubin Ghahramani and Kristian Kersting and Isabel Valera},
  title = {{Automatic Bayesian Density Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33015207},
}
Figure