Abstract
Irregular distribution in latent space causes posterior collapse, misalignment between posterior and prior, and ill-sampling problem in Variational Autoencoders (VAEs). In this paper, we introduce a novel adaptable three-stage Uniform Transformation (UT) module — Gaussian Kernel Density Estimation (G-KDE) clustering, non-parametric Gaussian Mixture (GM) Modeling, and Probability Integral Transform (PIT) — to address irregular latent distributions. By reconfiguring irregular distributions into a uniform distribution in the latent space, our approach significantly enhances the disentanglement and interpretability of latent representations, overcoming the limitation of traditional VAE models in capturing complex data structures. Empirical evaluations demonstrated the efficacy of our proposed UT module in improving disentanglement metrics across benchmark datasets — dSprites and MNIST. Our findings suggest a promising direction for advancing representation learning techniques, with implication for future research in extending this framework to more sophisticated datasets and downstream tasks.
Introduction
Variational Autoencoders (VAEs) are designed to compress high-dimensional data into a compact latent space. Ideally, this space should be smooth and easy to sample from, which is why standard VAEs typically regularize latent variables toward a standard Gaussian distribution. However, real-world data features — such as categorical shapes or periodic rotations — often cause the latent space to "clump" into irregular Gaussian Mixtures (GMs) rather than a single smooth bell curve.
These irregularities lead to three primary challenges:
- Posterior Collapse: Some latent dimensions are ignored entirely by the model.
- Distribution Misalignment: The encoder’s output does not match the prior distribution expected by the decoder.
- Ill-sampling: When the decoder samples from "empty" areas between clusters, it produces blurry or invalid results.
The Uniform Transformation (UT) Module
To mitigate these issues, we propose a three-stage Uniform Transformation (UT) module that acts as a bridge between the encoder and decoder, refining the latent space without requiring external labels.
- Stage 1: G-KDE Clustering: The module uses Gaussian Kernel Density Estimation to automatically identify clusters in each latent dimension. This non-parametric approach detects the number of modes without prior specification.
- Stage 2: GM Modeling: The identified clusters are formalized into a mathematical Gaussian Mixture model, capturing the weights, means, and variances of the underlying child distributions.
- Stage 3: Probability Integral Transform (PIT): This core step uses the Cumulative Distribution Function (CDF) of the learned GM to map irregular values into a uniform distribution. This "densifies" the representation, filling gaps and creating a bounded space more suitable for periodic data like rotation.
Experimental Evaluation
The UT module was evaluated on the dSprites and MNIST benchmarks. On the dSprites dataset, adding the UT module to a standard $\beta$-VAE increased the disentanglement metric from 0.62 to 0.90. Qualitatively, the VAE+UT model produced significantly sharper reconstructions, particularly for complex shapes like hearts, which often appear blurry in standard VAE outputs.
By mapping periodic features into a bounded uniform space, the representation becomes more natural, avoiding the issues associated with the infinite tails of a Gaussian distribution. This self-mining process improves the purity of latent variables by making better use of the information already captured by the encoder.
Conclusion and Future Work
The proposed UT module effectively reconfigures irregular latent distributions into structured uniform representations, significantly boosting disentanglement and interpretability. Future research will focus on addressing edge artifacts observed during transformation and scaling the framework to more complex, high-dimensional datasets beyond basic 2D shapes and digits.
Summary updated