General Table Completion using a Bayesian Nonparametric Model

Isabel Valera, Zoubin GhahramaniNeurIPS 2014 · OpenReview

Introduction

Most machine learning models for "table completion" (imputing missing values) assume that a database is homogeneous—meaning all columns contain the same type of data, usually modeled as continuous Gaussian distributions. However, real-world databases are inherently heterogeneous. A single table might include a person’s age (count data), gender (categorical), salary (positive real-valued), and survey responses (ordinal). Forcing these diverse types into a standard Gaussian model often yields inaccurate or physically impossible estimates, such as predicting a negative value for age.

To address this lack of principled, probabilistic tools for mixed-type data, researchers Isabel Valera and Zoubin Ghahramani propose a robust framework based on the Indian Buffet Process (IBP). Their model can simultaneously handle continuous, categorical, ordinal, and count data, providing a highly accurate table completion tool that scales linearly with the number of observations.

The General Observation Model

The foundation of this approach is the Indian Buffet Process (IBP), a Bayesian nonparametric prior. In the IBP, each object (or row in a database) is represented by a set of latent binary features. The model is nonparametric, meaning it does not require the user to pre-specify the number of latent features; instead, it learns how many features are needed to explain the data.

The core innovation of the paper is the General Observation Model, which bridges the gap between these binary latent features and diverse real-world data types.

Graphical model of the General IBP

To achieve this, the authors introduce auxiliary Gaussian variables ($y$) as an intermediate layer.

  1. Latent Mapping: A hidden Gaussian variable $y$ is modeled as a linear combination of the binary features.
  2. Transformation Functions: Specific mathematical transformations are then applied to $y$ to produce the final observed data type $x$:
  3. Real-valued: $x = y$.
  4. Positive real: A monotonic function ensures the output is strictly positive.
  5. Categorical: A "probit" model selects the category based on which auxiliary variable is the largest.
  6. Ordinal: The real line is divided into regions by thresholds; the region $y$ falls into determines the ordinal value.
  7. Count: The floor of a transformation function applied to $y$.

This "Gaussian Bridge" elegantly preserves the mathematical convenience of Gaussian distributions while accurately outputting non-Gaussian data types. To make the model practical for large datasets, the authors employ an accelerated Gibbs sampler for inference, ensuring computation time grows linearly ($O(N)$) with the number of objects.

Experimental Results

The model, termed the General IBP (GIBP), was evaluated on five diverse, real-world datasets: Statlog German Credit, QSAR Biodegradation, Internet Usage Survey, Wine Quality, and the NESARC personality disorder database.

Predictive log-likelihood across five real-world datasets

The results demonstrate the brittleness of treating heterogeneous data as purely Gaussian. The GIBP provided stable, superior estimates across all attributes.

  • Superior Accuracy: In four out of five datasets, the GIBP outperformed both the standard Gaussian IBP (SIBP) and Bayesian Probabilistic Matrix Factorization (BPMF) in terms of predictive log-likelihood.
  • Handling Sparsity: The model remained robust even when up to 50% (and sometimes more) of the data was missing.
  • Feature Discovery: The model successfully inferred a manageable number of latent features (typically under 25-50) to explain complex databases containing thousands of entries.
  • Scalability: The linear scaling of the inference algorithm allowed it to efficiently handle the NESARC dataset, which contains over 43,000 instances.

Limitations and Future Directions

While powerful, the current framework assumes that missing data is Missing Completely At Random (MCAR). In many real-world scenarios, data is "Missing Not At Random" (e.g., individuals with high salaries might be less likely to report them). The model does not yet account for these inherent biases. Additionally, for positive real and count data, the model relies on a transformation function with a user-defined parameter, meaning results can be sensitive to this hyper-parameter choice.

Looking forward, the authors suggest this framework could be extended beyond simple imputation. Its robust probabilistic foundation makes it a strong candidate for detecting errors in databases (outlier detection) or providing probabilistic responses to complex database queries.

Summary updated

Abstract

Even though heterogeneous databases can be found in a broad variety of applications, there exists a lack of tools for estimating missing data in such databases. In this paper, we provide an efficient and robust table completion tool, based on a Bayesian nonparametric latent feature model. In particular, we propose a general observation model for the Indian buffet process (IBP) adapted to mixed continuous (real-valued and positive real-valued) and discrete (categorical, ordinal and count) observations. Then, we propose an inference algorithm that scales linearly with the number of observations. Finally, our experiments over five real databases show that the proposed approach provides more robust and accurate estimates than the standard IBP and the Bayesian probabilistic matrix factorization with Gaussian observations.
BibTeX
@inproceedings{valera2014general,
  author = {Isabel Valera and Zoubin Ghahramani},
  title = {{General Table Completion using a Bayesian Nonparametric Model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=r1VDx_Z_-H},
}
Figure