General Latent Feature Models for Heterogeneous Datasets

Isabel Valera, Melanie F. Pradier, María Lomelí, Zoubin Ghahramani

Abstract

Latent feature modeling allows capturing the latent structure responsible for generating the observed properties of a set of objects. It is often used to make predictions either for new values of interest or missing information in the original data, as well as to perform data exploratory analysis. However, although there is an extensive literature on latent feature models for homogeneous datasets, where all the attributes that describe each object are of the same (continuous or discrete) nature, there is a lack of work on latent feature modeling for heterogeneous databases. In this paper, we introduce a general Bayesian nonparametric latent feature model suitable for heterogeneous datasets, where the attributes describing each object can be either discrete, continuous or mixed variables. The proposed model presents several important properties. First, it accounts for heterogeneous data while keeping the properties of conjugate models, which allow us to infer the model in linear time with respect to the number of objects and attributes. Second, its Bayesian nonparametric nature allows us to automatically infer the model complexity from the data, i.e., the number of features necessary to capture the latent structure in the data. Third, the latent features in the model are binary-valued variables, easing the interpretability of the obtained latent features in data exploratory analysis. We show the flexibility of the proposed model by solving both prediction and data analysis tasks on several real-world datasets. Moreover, a software package of the GLFM is publicly available for other researcher to use and improve it.

Introduction

Discovering hidden variables that explain observed data is a cornerstone of machine learning. However, most existing latent feature models assume data homogeneity—expecting either all continuous or all discrete variables. Real-world datasets, such as Electronic Health Records, are inherently heterogeneous, containing a mix of continuous (e.g., blood pressure), categorical (e.g., diagnosis codes), ordinal (e.g., pain scales), and count data (e.g., hospital visits). Forcing these diverse types into a single Gaussian mold often leads to information loss and poor interpretability. This paper introduces the General Latent Feature Model (GLFM), a unified Bayesian framework designed to handle mixed data types while maintaining computational efficiency and automatic model selection.

Methodology

The GLFM integrates two key components to handle heterogeneity and complexity:

1. The Indian Buffet Process (IBP)

The model utilizes the IBP as a Bayesian nonparametric prior. This allows the model to automatically determine the number of latent features ($K$) required to represent the data, avoiding the need for manual tuning or cross-validation for model complexity. Objects are represented as combinations of binary latent features, which significantly enhances the interpretability of exploratory data analysis.

2. The Pseudo-Observation Trick

To manage diverse data types while preserving mathematical tractability, the authors introduce an auxiliary Gaussian variable $y_n^d$, termed a "pseudo-observation." The model assumes that binary latent features generate this hidden Gaussian value, which is subsequently transformed via a mapping function $f_d(\cdot)$ to produce the observed data $x_n^d$. This approach allows the derivation of specific likelihoods for continuous, categorical, ordinal, and count data types.

Graphical representation of the General Latent Feature Model (GLFM) showing the relationship between latent features Z, pseudo-observations Y, and observed data X.

Inference and Scalability

For model estimation, the authors employ an Accelerated Gibbs Sampler. By linking non-Gaussian observations to a hidden Gaussian layer, the model preserves the properties of conjugate models. This ensures that the inference algorithm scales linearly $O(N)$ with the number of objects, making it suitable for large-scale datasets.

The inference algorithm for GLFM, detailing the iterative sampling of latent features and pseudo-observations.

Experimental Results

The GLFM was evaluated across several real-world datasets, including financial credit data, chemical descriptors, and clinical trial results.

  • Missing Data Estimation: In table completion tasks, GLFM consistently outperformed Bayesian Probabilistic Matrix Factorization (BPMF) and standard linear-Gaussian IBP (SIBP). It achieved significantly higher log-likelihoods, particularly on discrete and count attributes where Gaussian assumptions are inappropriate.
  • Clinical and Demographic Insights: On prostate cancer data, the model identified a latent feature linking high dosages of the drug DES to increased cardiovascular mortality—a known clinical side effect. On US election data, it successfully clustered counties into meaningful profiles (e.g., "Big Cities" vs. "Rural Farming") that aligned with historical voting patterns.

Conclusion

The GLFM provides a robust solution for modeling messy, heterogeneous datasets. By leveraging binary latent features and a pseudo-observation framework, it offers both high predictive performance and clear interpretability. While the model may require preprocessing for extremely heavy-tailed distributions, its ability to automatically infer complexity and handle mixed data types marks a significant advancement in latent feature modeling. To support the research community, the authors have released the GLFM toolbox in Python, Matlab, and R.

Summary updated

Abstract

Latent feature modeling allows capturing the latent structure responsible for generating the observed properties of a set of objects. It is often used to make predictions either for new values of interest or missing information in the original data, as well as to perform data exploratory analysis. However, although there is an extensive literature on latent feature models for homogeneous datasets, where all the attributes that describe each object are of the same (continuous or discrete) nature, there is a lack of work on latent feature modeling for heterogeneous databases. In this paper, we introduce a general Bayesian nonparametric latent feature model suitable for heterogeneous datasets, where the attributes describing each object can be either discrete, continuous or mixed variables. The proposed model presents several important properties. First, it accounts for heterogeneous data while keeping the properties of conjugate models, which allow us to infer the model in linear time with respect to the number of objects and attributes. Second, its Bayesian nonparametric nature allows us to automatically infer the model complexity from the data, i.e., the number of features necessary to capture the latent structure in the data. Third, the latent features in the model are binary-valued variables, easing the interpretability of the obtained latent features in data exploratory analysis. We show the flexibility of the proposed model by solving both prediction and data analysis tasks on several real-world datasets. Moreover, a software package of the GLFM is publicly available for other researcher to use and improve it.
BibTeX
@article{valera2020general,
  author = {Isabel Valera and Melanie F. Pradier and María Lomelí and Zoubin Ghahramani},
  title = {{General Latent Feature Models for Heterogeneous Datasets}},
  journal = {J. Mach. Learn. Res.},
  year = {2020},
  url = {https://openreview.net/forum?id=dRo80L6s5g},
}
Figure