Uniform Transformation: Refining Latent Representation in Variational Autoencoders

Ye Shi, C. S. George Lee

Abstract

Irregular distribution in latent space causes posterior collapse, misalignment between posterior and prior, and ill-sampling problem in Variational Autoencoders (VAEs). In this paper, we introduce a novel adaptable three-stage Uniform Transformation (UT) module — Gaussian Kernel Density Estimation (G-KDE) clustering, non-parametric Gaussian Mixture (GM) Modeling, and Probability Integral Transform (PIT) — to address irregular latent distributions. By reconfiguring irregular distributions into a uniform distribution in the latent space, our approach significantly enhances the disentanglement and interpretability of latent representations, overcoming the limitation of traditional VAE models in capturing complex data structures. Empirical evaluations demonstrated the efficacy of our proposed UT module in improving disentanglement metrics across benchmark datasets — dSprites and MNIST. Our findings suggest a promising direction for advancing representation learning techniques, with implication for future research in extending this framework to more sophisticated datasets and downstream tasks.

Introduction

Variational Autoencoders (VAEs) are designed to compress high-dimensional data into a compact latent space. Ideally, this space should be smooth and easy to sample from, which is why standard VAEs typically regularize latent variables toward a standard Gaussian distribution. However, real-world data features — such as categorical shapes or periodic rotations — often cause the latent space to "clump" into irregular Gaussian Mixtures (GMs) rather than a single smooth bell curve.

These irregularities lead to three primary challenges:

  1. Posterior Collapse: Some latent dimensions are ignored entirely by the model.
  2. Distribution Misalignment: The encoder’s output does not match the prior distribution expected by the decoder.
  3. Ill-sampling: When the decoder samples from "empty" areas between clusters, it produces blurry or invalid results.
Latent variables in standard VAEs often converge into irregular mixture patterns rather than smooth Gaussians, as seen in these histograms of latent dimensions.

The Uniform Transformation (UT) Module

To mitigate these issues, we propose a three-stage Uniform Transformation (UT) module that acts as a bridge between the encoder and decoder, refining the latent space without requiring external labels.

  1. Stage 1: G-KDE Clustering: The module uses Gaussian Kernel Density Estimation to automatically identify clusters in each latent dimension. This non-parametric approach detects the number of modes without prior specification.
  2. Stage 2: GM Modeling: The identified clusters are formalized into a mathematical Gaussian Mixture model, capturing the weights, means, and variances of the underlying child distributions.
  3. Stage 3: Probability Integral Transform (PIT): This core step uses the Cumulative Distribution Function (CDF) of the learned GM to map irregular values into a uniform distribution. This "densifies" the representation, filling gaps and creating a bounded space more suitable for periodic data like rotation.
The UT module pipeline: Stage 1 (G-KDE Clustering), Stage 2 (GM Modeling), and Stage 3 (PIT) transforming irregular latents into a uniform space.

Experimental Evaluation

The UT module was evaluated on the dSprites and MNIST benchmarks. On the dSprites dataset, adding the UT module to a standard $\beta$-VAE increased the disentanglement metric from 0.62 to 0.90. Qualitatively, the VAE+UT model produced significantly sharper reconstructions, particularly for complex shapes like hearts, which often appear blurry in standard VAE outputs.

By mapping periodic features into a bounded uniform space, the representation becomes more natural, avoiding the issues associated with the infinite tails of a Gaussian distribution. This self-mining process improves the purity of latent variables by making better use of the information already captured by the encoder.

Conclusion and Future Work

The proposed UT module effectively reconfigures irregular latent distributions into structured uniform representations, significantly boosting disentanglement and interpretability. Future research will focus on addressing edge artifacts observed during transformation and scaling the framework to more complex, high-dimensional datasets beyond basic 2D shapes and digits.

Summary updated

Abstract

Irregular distribution in latent space causes posterior collapse, misalignment between posterior and prior, and ill-sampling problem in Variational Autoencoders (VAEs). In this paper, we introduce a novel adaptable three-stage Uniform Transformation (UT) module -- Gaussian Kernel Density Estimation (G-KDE) clustering, non-parametric Gaussian Mixture (GM) Modeling, and Probability Integral Transform (PIT) -- to address irregular latent distributions. By reconfiguring irregular distributions into a uniform distribution in the latent space, our approach significantly enhances the disentanglement and interpretability of latent representations, overcoming the limitation of traditional VAE models in capturing complex data structures. Empirical evaluations demonstrated the efficacy of our proposed UT module in improving disentanglement metrics across benchmark datasets -- dSprites and MNIST. Our findings suggest a promising direction for advancing representation learning techniques, with implication for future research in extending this framework to more sophisticated datasets and downstream tasks.
BibTeX
@misc{shi2024uniform,
  author = {Ye Shi and C. S. George Lee},
  title = {{Uniform Transformation: Refining Latent Representation in Variational Autoencoders}},
  year = {2024},
  eprint = {2407.02681},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.02681},
}
Figure