Automatic Bayesian Density Analysis

Antonio Vergari, Alejandro Molina, Robert Peharz, Zoubin Ghahramani, Kristian Kersting, Isabel Valera

Abstract

Exploratory Data Analysis (EDA) is a fundamental yet labor-intensive stage in machine learning. Automatic Bayesian Density Analysis (ABDA) introduces a "Swiss Army knife" approach to unsupervised EDA by integrating Sum-Product Networks (SPNs) into a fully Bayesian framework. ABDA automates the discovery of statistical data types, likelihood model selection, missing value imputation, and anomaly detection in heterogeneous tabular data, significantly reducing the need for manual statistical intervention.

Introduction

The initial phase of any data science project involves understanding the underlying structure of a dataset. However, domain experts often lack the statistical expertise to select appropriate models, and professional statisticians are a limited resource. Existing automated tools frequently suffer from rigidity—assuming Gaussian or categorical distributions—or brittleness when faced with missing values and outliers. Furthermore, deep probabilistic models can be computationally intractable for high-dimensional data.

ABDA addresses these challenges by providing a robust, automated tool for making sense of "messy" real-world tabular data. It treats the statistical type of a variable as a latent variable, allowing the model to learn whether a feature is Gaussian, Gamma, Poisson, or categorical based on the data itself.

Overview of the ABDA process: (a) input data with missing values, (b) discovered clusters and types, (c) local densities, (d) type distributions, and (e) extracted patterns.

Methodology: Type-Augmented Sum-Product Networks

The core of ABDA is a hierarchical probabilistic model built on Sum-Product Networks (SPNs). SPNs are deep graphical models that allow for tractable exact inference.

Hierarchical Backbone

ABDA utilizes an SPN structure to perform "hierarchical co-clustering." It recursively partitions data into independent groups of features (via product nodes) and clusters of similar samples (via sum nodes).

Likelihood Dictionaries and Bayesian Inference

Unlike standard SPNs with fixed leaf distributions, ABDA equips each leaf with a "dictionary" of potential models (e.g., Gaussian, Gamma, Poisson). The model employs Gibbs sampling to maintain uncertainty over both the global dependency structure and the local data types. This Bayesian approach allows the system to learn a posterior distribution over types rather than forcing a single, potentially incorrect, distribution.

The Type-augmented SPN architecture, showing how sum and product nodes lead to leaf distributions selected from a dictionary of types.

Structure Learning

Initial dependencies are identified using the Randomized Dependence Coefficient (RDC), which is likelihood-agnostic. These dependencies are subsequently refined through the Bayesian learning process.

Experimental Evaluation

The authors evaluated ABDA across three domains:

  1. Synthetic Data: 90 datasets were used to verify the model's ability to recover ground-truth distributions and dependency structures.
  2. Real-World Benchmarks: 12 datasets (e.g., Abalone, Adult, Wine Quality) were used for density estimation and imputation tasks, comparing ABDA against Mixed Sum-Product Networks (MSPNs) and Latent Variable Matrix Factorization (ISLV).
  3. Anomaly Detection: 10 standard datasets (e.g., Kdd99, Shuttle) were used to test ABDA against specialized algorithms like One-Class SVM and Local Outlier Factor (LOF).

Key Results

  • Density Estimation: ABDA consistently outperformed baselines in test log-likelihood. It proved particularly effective in inductive settings, where MSPNs often overfit.
  • Type Recovery: On synthetic data, ABDA achieved near-perfect accuracy in distinguishing between similar distributions (e.g., Gaussian vs. Gamma).
  • Anomaly Detection: Despite being a general-purpose estimator, ABDA matched or exceeded the performance of specialized state-of-the-art anomaly detectors on 7 out of 10 benchmarks.
  • Interpretability: The model can extract human-readable "conjunctive patterns"—logical rules that describe specific sub-populations within the data.

Discussion and Key Insights

A significant insight from ABDA is the treatment of "statistical type" as an inferable latent variable. This allows the model to recognize that a variable's behavior may change across different contexts; for instance, a feature might be Gaussian in one cluster but highly skewed in another. Furthermore, the model's structure naturally handles anomalies by assigning them to low-weighted mixture components, preventing them from skewing the overall model parameters.

While powerful, the Bayesian Gibbs sampling approach is more computationally intensive than maximum-likelihood training. Future work aims to automate prior selection and integrate natural language generation to transform density analysis into automated data storytelling.

Summary updated

Abstract

Making sense of a dataset in an automatic and unsupervised fashion is a challenging problem in statistics and AI. Classical approaches for exploratory data analysis are usually not flexible enough to deal with the uncertainty inherent to real-world data: they are often restricted to fixed latent interaction models and homogeneous likelihoods; they are sensitive to missing, corrupt and anomalous data; moreover, their expressiveness generally comes at the price of intractable inference. As a result, supervision from statisticians is usually needed to find the right model for the data. However, since domain experts are not necessarily also experts in statistics, we propose Automatic Bayesian Density Analysis (ABDA) to make exploratory data analysis accessible at large. Specifically, ABDA allows for automatic and efficient missing value estimation, statistical data type and likelihood discovery, anomaly detection and dependency structure mining, on top of providing accurate density estimation. Extensive empirical evidence shows that ABDA is a suitable tool for automatic exploratory analysis of mixed continuous and discrete tabular data.
BibTeX
@inproceedings{vergari2019automatic,
  author = {Antonio Vergari and Alejandro Molina and Robert Peharz and Zoubin Ghahramani and Kristian Kersting and Isabel Valera},
  title = {{Automatic Bayesian Density Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33015207},
}
Figure