XPCA: Extending PCA for a Combination of Discrete and Continuous Variables

Clifford Anderson-Bergman, Tamara G. Kolda, Kina Kincher-WinotoarXiv 2018 · arXiv

Introduction

Principal Component Analysis (PCA) is a cornerstone of data science, widely used for dimensionality reduction and missing value imputation. However, its effectiveness is fundamentally tied to the assumption that data follows a Gaussian (normal) distribution. In real-world applications—such as analyzing NBA player statistics or legislative voting records—datasets are often "heterogeneous," comprising a mix of continuous, binary, and ordinal variables. Standard PCA often fails in these contexts, producing nonsensical predictions like negative probabilities for binary events.

The Challenge of Heterogeneous Data

While extensions like Copula Component Analysis (COCA) have improved PCA's performance on non-normal continuous data, they struggle with discrete variables. Discrete data introduces "jumps" or "heavy atoms" (values with high probability, such as zero counts) that violate the continuity assumptions of standard copula methods.

Applicability of PCA, COCA, and XPCA across various data distributions, highlighting XPCA's robustness to zero-inflated and discrete data.

Methodology: The XPCA Framework

XPCA (Extended Principal Component Analysis) addresses these limitations through two primary innovations:

  1. Gaussian Copulas: By employing a copula, XPCA decouples the dependency structure (correlations) from the marginal distributions of individual variables. This allows the model to accommodate diverse data shapes within a unified framework.
  2. Interval Likelihood: This is the core advancement. Rather than mapping discrete values to single points in a latent space, XPCA treats them as intervals. It calculates the likelihood of an observation by integrating the Gaussian density over a specific range. This approach mathematically accounts for the discontinuities inherent in discrete and zero-inflated data.

The model is optimized using maximum likelihood estimation, supported by efficient algorithms like L-BFGS and block coordinate descent.

Experimental Evaluation

The authors validated XPCA across three distinct scenarios:

  • Simulated Data: Testing on mixed binary and Gaussian distributions showed that XPCA significantly outperformed both PCA and COCA in Mean Squared Error (MSE).
  • NBA Player Statistics: Analyzing a dataset of 40 variables for 476 players, XPCA achieved a 4–8% improvement in MSE. Crucially, it was the only method to accurately capture correlations involving rare events, such as "Triple Doubles."
  • U.S. Senate Voting Records: In a highly sparse dataset (63% missing), XPCA demonstrated superior stability. While standard PCA overfitted as model complexity increased, XPCA maintained a consistent representation of political structures and bipartisan blocks.

Key Insights

  • Range-Respecting Imputation: XPCA's probabilistic nature ensures that imputed values respect the natural bounds of the data (e.g., probabilities between 0 and 1).
  • Handling "Heavy Atoms": The interval mapping allows XPCA to gracefully handle "walls" of zeros or other frequent discrete values that typically confound standard algorithms.

Conclusion and Future Work

XPCA provides a robust framework for analyzing the messy, heterogeneous data typical of real-world applications. While it is computationally more intensive than standard PCA and currently lacks support for unordered categorical data, it represents a significant step forward in making latent variable models more versatile and accurate. Future research may focus on optimizing computational speed and extending the model to include column-specific noise parameters.

Summary updated

Abstract

Principal component analysis (PCA) is arguably the most popular tool in multivariate exploratory data analysis. In this paper, we consider the question of how to handle heterogeneous variables that include continuous, binary, and ordinal. In the probabilistic interpretation of low-rank PCA, the data has a normal multivariate distribution and, therefore, normal marginal distributions for each column. If some marginals are continuous but not normal, the semiparametric copula-based principal component analysis (COCA) method is an alternative to PCA that combines a Gaussian copula with nonparametric marginals. If some marginals are discrete or semi-continuous, we propose a new extended PCA (XPCA) method that also uses a Gaussian copula and nonparametric marginals and accounts for discrete variables in the likelihood calculation by integrating over appropriate intervals. Like PCA, the factors produced by XPCA can be used to find latent structure in data, build predictive models, and perform dimensionality reduction. We present the new model, its induced likelihood function, and a fitting algorithm which can be applied in the presence of missing data. We demonstrate how to use XPCA to produce an estimated full conditional distribution for each data point, and use this to produce to provide estimates for missing data that are automatically range respecting. We compare the methods as applied to simulated and real-world data sets that have a mixture of discrete and continuous variables.
BibTeX
@misc{andersonbergman2018xpca,
  author = {Clifford Anderson-Bergman and Tamara G. Kolda and Kina Kincher-Winoto},
  title = {{XPCA: Extending PCA for a Combination of Discrete and Continuous Variables}},
  year = {2018},
  eprint = {1808.07510},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.07510},
}
Figure