Dimension Reduction for High-dimensional Small Counts with KL Divergence

Yurong Ling, Jing-Hao XueUAI 2022 · OpenReview

Abstract

Dimension reduction for high-dimensional count data with a large proportion of zeros is an important task in various applications. As a large number of dimension reduction methods rely on the proximity measure, we develop a dissimilarity measure that is well-suited for small counts based on the Kullback-Leibler divergence. We compare the proposed measure with other widely used dissimilarity measures and show that the proposed one has superior discrimination ability when applied to high-dimensional count data having an excess of zeros. Extensive empirical results, on both simulated and publicly-available real-world datasets that contain many zeros, demonstrate that the proposed dissimilarity measure can improve a wide range of dimension reduction methods.

Introduction

In fields such as genomics and ecology, researchers frequently encounter high-dimensional count data characterized by a high proportion of zeros. Analyzing such data often requires Dimension Reduction (DR) to project high-dimensional features into a lower-dimensional space for visualization and clustering. The conventional approach involves applying a Variance-Stabilizing Transformation (VST), such as the logarithm or square root, followed by Principal Component Analysis (PCA). However, these transformations are primarily designed for large counts and often fail to stabilize variance or preserve meaningful structures in "small count" data with an excess of zeros. Standard DR methods typically assume a Gaussian distribution, which is fundamentally mismatched with the discrete, skewed nature of sparse count data.

Methodology

To address these limitations, this research proposes a proximity measure specifically designed for the statistical properties of small counts. Instead of geometric distances like Euclidean distance, the proposed approach treats each data point as a realization of a probability distribution and utilizes the Kullback-Leibler (KL) divergence to quantify dissimilarity.

Dissimilarity Measures

Two primary measures are derived: $D_P$, based on the Poisson distribution, and $D_{NB}$, based on the Negative Binomial distribution. $D_{NB}$ is particularly effective as it accounts for overdispersion—a common characteristic in biological datasets where the variance exceeds the mean.

Empirical Bayes Estimation

Estimating parameters for individual data points with many zeros can be numerically unstable. To overcome this, the authors employ Empirical Bayes estimators. This method uses the global mean of a feature as a prior to smooth estimates for individual points, ensuring robust pairwise distance calculations even in the presence of extreme sparsity.

Results and Discussion

The proposed measures were evaluated using single-cell RNA sequencing (scRNA-seq) data, including five real-world datasets and three simulated datasets generated to mimic genomic properties.

Visualization and Clustering Performance

Visual inspection of 2D projections demonstrates that $D_{NB}$ produces significantly more distinct and tighter clusters compared to standard Euclidean or VST-based distances. In many instances, $D_{NB}$ was the only measure capable of resolving specific cell types that appeared as overlapping clusters under other methods.

Comparison of 2D projections for different dissimilarity measures. $D_{NB}$ (bottom right) shows the clearest separation of clusters.

Quantitative evaluation using the Adjusted Rand Index (ARI) further confirms the superiority of $D_{NB}$. Across various datasets, $D_{NB}$ consistently outperformed Euclidean distance ($D_E$) and common transformations like the log-transform ($D_{log}$). For example, on the sim-manno-vm dataset, $D_{NB}$ achieved an ARI near 0.9, while standard methods remained around 0.2.

Adjusted Rand Index (ARI) scores across eight datasets (a-h). The proposed $D_{NB}$ measure (red bar) consistently achieves the highest clustering accuracy.

Conclusion

This paper introduces a novel dissimilarity measure, $D_{NB}$, that leverages KL divergence and Negative Binomial modeling to improve dimension reduction for sparse count data. By moving away from forced Gaussian transformations and instead respecting the underlying discrete distribution of the data, $D_{NB}$ provides more accurate visualizations and clustering results. Future work includes integrating these measures into deep learning frameworks, such as Variational Autoencoders, to further enhance high-dimensional data analysis.

Summary updated

Abstract

Dimension reduction for high-dimensional count data with a large proportion of zeros is an important task in various applications. As a large number of dimension reduction methods rely on the proximity measure, we develop a dissimilarity measure that is well-suited for small counts based on the Kullback-Leibler divergence. We compare the proposed measure with other widely used dissimilarity measures and show that the proposed one has superior discrimination ability when applied to high-dimensional count data having an excess of zeros. Extensive empirical results, on both simulated and publicly-available real-world datasets that contain many zeros, demonstrate that the proposed dissimilarity measure can improve a wide range of dimension reduction methods.
BibTeX
@inproceedings{ling2022dimension,
  author = {Yurong Ling and Jing-Hao Xue},
  title = {{Dimension Reduction for High-dimensional Small Counts with KL Divergence}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=BhzEFwLj5l5},
}
Figure