Learning Manifold Dimensions with Conditional Variational Autoencoders

Yijia Zheng, Tong He, Yixuan Qiu, David Wipf

TL;DR

This paper provides the first rigorous proof that the global minima of Variational Autoencoders (VAEs) and their conditional counterparts (CVAEs) correctly recover the underlying dimension of a data manifold. The authors demonstrate that conditioning variables can adaptively reduce the number of active latent dimensions and warn that common practices, like weight sharing in sequential models, can actually prevent these models from learning the true manifold structure.

What Problem Does This Paper Address?

In machine learning, we often assume the "manifold hypothesis": high-dimensional data, like images, actually lie on or near a much lower-dimensional surface (a manifold). For a generative model like a VAE to be truly effective, it should ideally learn the correct dimensionality of this surface.

While researchers have observed that VAEs seem to "prune" unnecessary latent dimensions, this behavior lacked a rigorous theoretical foundation, especially for Conditional VAEs (CVAEs). Furthermore, real-world data often consists of a union of manifolds (e.g., the manifold for the digit "1" is likely simpler than the manifold for "8"). This paper sets out to prove that VAEs/CVAEs are theoretically capable of recovering these dimensions and investigates how practical design choices—like decoder variance handling and weight sharing—impact this ability.

Methods

The authors define a $\kappa$-simple VAE/CVAE, a standardized model architecture with Gaussian encoders, decoders, and priors, where the mean functions are Lipschitz continuous.

Their theoretical approach centers on analyzing the VAE loss function (the ELBO) as the decoder variance ($\gamma$) tends toward zero. They introduce the concept of Active Latent Dimensions: a latent dimension is "active" if its corresponding encoder variance shrinks to zero as the decoder noise vanishes.

Key theoretical contributions include:

  • Theorem 1: Proves that for a VAE, the number of active latent dimensions at a global minimum almost surely equals the true manifold dimension $r$.
  • Theorem 2: Extends this to CVAEs, showing that if a conditioning variable $c$ has an "effective dimension" $t$, the number of active latent dimensions becomes $r - t$.
  • Theorem 3: Analyzes sequential CVAEs, proving that sharing weights between the prior and the encoder (a common practice) creates a lower bound on the loss that prevents the model from learning low-dimensional manifolds.

Datasets

The authors validate their theory using:

  • Synthetic Data: Low-dimensional Gaussian mixtures projected into high-dimensional space via random matrices and sigmoid activations. This allows for precise control over the ground-truth manifold dimension ($r$) and the conditioning information ($t$).
  • MNIST & Fashion-MNIST: Standard image datasets used to test the model's behavior on real-world data where the manifold structure is intuitive but not explicitly known.

Key Insights and Ideas

  • Automatic Dimensionality Selection: The VAE loss surface is shaped such that global minimizers naturally "shut off" superfluous latent dimensions. This isn't just a heuristic; it's a fundamental property of the optimization objective.
  • Conditioning as a "Free Lunch": In CVAEs, conditioning variables can replace latent dimensions. Because using the conditioning variable to reconstruct data doesn't incur a KL-divergence penalty (unlike latent variables), the model prefers to use $c$ as much as possible, reducing the active latent count to $r - t$.
  • The Weight-Sharing Trap: In sequential models, the encoder and prior have different jobs. The encoder needs to be precise (low variance) to capture the manifold, while the prior needs to maintain enough randomness to match the data distribution's uncertainty. Forcing them to share weights makes it impossible to satisfy both requirements simultaneously, leading to poor manifold learning.
  • Full vs. Diagonal Covariance: The authors find that using a full covariance matrix for the encoder (rather than the standard diagonal one) helps the model avoid bad local minima and more reliably find the optimal sparse representation.

Key Results

The empirical results strongly support the theoretical proofs. As shown in Table 1, when the latent capacity $\kappa$ is greater than or equal to the true dimension $r$, the number of active dimensions (AD) perfectly matches $r$ as the decoder variance $\gamma$ decreases.

Table 1: Aligning learned VAE active dimensions (AD) with the ground-truth manifold dimension r. When κ ≥r (a surplus of latent dimensions) the VAE largely succeeds as γ, the reconstruction error (Recon), and -ELBO converge to relatively small values; however, when κ < r this is not possible.

On real datasets like MNIST, the model shows a clear saturation point for active dimensions. Even as the total latent capacity $\kappa$ increases from 16 to 32, the number of active dimensions remains relatively stable (around 12-13 for MNIST), suggesting the model has found the "true" intrinsic dimensionality of the data.

Table 2: VAE latent compression on real datasets in further support of Theorem 1.

Limitations and Future Work

The primary limitation is that the proofs rely on finding global minima. In practice, deep learning models often settle in local minima. While the authors show that certain choices (like full covariance encoders) help, there is still a gap between the "provable" global optimum and the "reachable" local one.

Additionally, the authors note that while they introduced an attention mechanism to help learn a union of manifolds, they haven't yet fully explored which specific decoder inductive biases (architectural choices) best align with different types of real-world manifold data. This remains a promising direction for future research.

Summary updated

Abstract

Although the variational autoencoder (VAE) and its conditional extension (CVAE) are capable of state-of-the-art results across multiple domains, their precise behavior is still not fully understood, particularly in the context of data (like images) that lie on or near a low-dimensional manifold. For example, while prior work has suggested that the globally optimal VAE solution can learn the correct manifold dimension, a necessary (but not sufficient) condition for producing samples from the true data distribution, this has never been rigorously proven. Moreover, it remains unclear how such considerations would change when various types of conditioning variables are introduced, or when the data support is extended to a union of manifolds (e.g., as is likely the case for MNIST digits and related). In this work, we address these points by first proving that VAE global minima are indeed capable of recovering the correct manifold dimension. We then extend this result to more general CVAEs, demonstrating practical scenarios whereby the conditioning variables allow the model to adaptively learn manifolds of varying dimension across samples. Our analyses, which have practical implications for various CVAE design choices, are also supported by numerical results on both synthetic and real-world datasets.
BibTeX
@inproceedings{zheng2022learning,
  author = {Yijia Zheng and Tong He and Yixuan Qiu and David Wipf},
  title = {{Learning Manifold Dimensions with Conditional Variational Autoencoders}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=NqZUpAG9z6},
}
Figure