Learning Manifold Dimensions with Conditional Variational Autoencoders

Yijia Zheng, Tong He, Yixuan Qiu, David Wipf

Abstract

Variational Autoencoders (VAEs) are widely used for learning low-dimensional representations of high-dimensional data, yet theoretical understanding of whether they recover the true manifold dimensionality has remained incomplete. This paper provides a rigorous proof that the global minima of VAE and Conditional VAE (CVAE) loss functions occur exactly when the model recovers the true dimensionality of the data manifold. We introduce the $r-t$ rule, demonstrating that conditioning variables allow the model to adaptively reduce active latent dimensions. Empirical results on synthetic manifolds and real-world datasets (MNIST, Fashion-MNIST) validate these theoretical findings and offer insights into architectural choices like weight sharing and decoder variance.

Introduction

In generative modeling, the Manifold Hypothesis posits that high-dimensional data actually resides on a much lower-dimensional surface. While VAEs are effective at sampling from these manifolds, a fundamental question persists: Do they learn the correct number of dimensions? Previous research suggested VAEs might "shut off" redundant dimensions, but this was largely limited to linear decoders. This work extends that understanding to non-linear, complex architectures, explaining how adding conditions (labels or temporal context) fundamentally alters the mathematics of representation learning.

Theoretical Framework: The $r-t$ Rule

The authors define a $\kappa$-simple VAE/CVAE, where encoders and decoders are Lipschitz continuous functions. By analyzing the model as decoder variance ($\gamma$) approaches zero, they investigate Active Latent Dimensions—dimensions where the encoder's uncertainty shrinks to zero as precision increases.

The core theoretical contribution is Theorem 2, which establishes the $r - t$ Rule. It proves that a CVAE will learn exactly $r - t$ active dimensions, where $r$ is the manifold dimension and $t$ is the dimension of the information provided by the condition. This implies the model is "efficiently lazy": if a label provides 2D worth of information, the latent space requirements are reduced by exactly that amount.

Experimental Validation

Synthetic Manifolds

To test the $r-t$ rule, the authors used high-dimensional data generated via non-linear projections of low-dimensional Gaussian mixtures. This allowed for precise control over the ground-truth manifold dimension ($r$).

Active dimension recovery on synthetic manifolds. The model consistently identifies the true manifold dimension $r$ or $r-t$ when conditioned.

As shown in the synthetic results, when the true manifold dimension was $r=10$, the VAE identified exactly 10 active dimensions. Upon adding a condition with $t=5$, the CVAE correctly reduced its representation to 5 active dimensions, confirming the theoretical prediction.

Real-World Image Data

The theory was further validated using MNIST and Fashion-MNIST. On MNIST, a standard VAE utilized approximately 13 active dimensions. When conditioned on digit labels, this count dropped to 12, suggesting that the class label "explains away" roughly one dimension of the image's total variance.

Active dimensions (AD) for MNIST and Fashion-MNIST datasets across different latent space capacities ($\kappa$).

Architectural Implications

Beyond dimension recovery, the paper highlights two critical design choices:

  1. The Weight-Sharing Trap: In sequential models (like RNN-VAEs), sharing weights between the prior and encoder is mathematically harmful. The encoder must be "sharp" (low variance) to locate the manifold, while the prior must be "broad" to capture global uncertainty. Forcing them to share weights prevents optimal manifold representation.
  2. Decoder Variance ($\gamma$): The initialization of learnable decoder variance is a critical hyperparameter. If $\gamma$ is initialized too high, the model may converge to a local minimum, failing to detect the manifold's complexity and defaulting to a single active dimension.

Conclusion

This work provides the first formal proof that VAEs and CVAEs are theoretically capable of recovering the true dimensionality of data manifolds. The $r-t$ rule offers a new lens through which to view conditional generative models, suggesting that latent spaces are not just static bottlenecks but adaptive structures that respond to the information density of external conditions.

Summary updated

BibTeX
@inproceedings{zheng2022learning,
  author = {Yijia Zheng and Tong He and Yixuan Qiu and David Wipf},
  title = {{Learning Manifold Dimensions with Conditional Variational Autoencoders}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=NqZUpAG9z6},
}
Figure