Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding

Akira Nakagawa, Keizo Kato, Taiji Suzuki

Abstract

Variational autoencoder (VAE) estimates the posterior parameters (mean and variance) of latent variables corresponding to each input data. While it is used for many tasks, the transparency of the model is still an underlying issue. This paper provides a quantitative understanding of VAE property through the differential geometric and information-theoretic interpretations of VAE. According to the Rate-distortion theory, the optimal transform coding is achieved by using an orthonormal transform with PCA basis where the transform space is isometric to the input. Considering the analogy of transform coding to VAE, we clarify theoretically and experimentally that VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter. As a result, we can estimate the data probabilities in the input space from the prior, loss metrics, and corresponding posterior parameters, and further, the quantitative importance of each latent variable can be evaluated like the eigenvalue of PCA.

Introduction

Despite their widespread adoption in generative modeling and representation learning, Variational Autoencoders (VAEs) have long been treated as "black boxes." While the maximization of the Evidence Lower Bound (ELBO) is well-understood, several fundamental questions remain: What do latent variables represent in physical or geometric terms? What is the precise quantitative role of the $\beta$ parameter in $\beta$-VAEs? And how can we measure the "importance" of a specific latent dimension as concretely as we do with eigenvalues in Principal Component Analysis (PCA)?

This research addresses these gaps by employing differential geometry and Rate-Distortion (RD) theory to create a unified quantitative framework. The authors demonstrate that VAEs act as a non-linear version of PCA-based transform coding by mapping them to the framework of implicit isometric embeddings.

Methods: Implicit Isometric Embedding

The core contribution of this work is the theoretical proof that a VAE implicitly creates a space where the metric of the input data is preserved, provided the latent variables are scaled correctly. In geometry, an isometric embedding is a map that preserves distances.

The mapping between VAE and Implicit Isometric Embedding.

The authors derive a mathematical link between the VAE's latent variables ($\mu$) and an implicit isometric space ($y$): $$\frac{dy_j}{d\mu_j} = \frac{\sqrt{\beta/2}}{\sigma_j}$$

This relationship reveals that the learned posterior variance ($\sigma_j$) acts as a local scaling factor. When this scale is accounted for, the VAE essentially performs transform coding—the same principle used in JPEG compression—but utilizes non-linear neural networks instead of linear transforms. To ensure the theory applies across various data types, the authors utilize metric tensors ($G_x$), allowing different reconstruction losses (e.g., MSE, Binary Cross-Entropy, or SSIM) to be treated under a single mathematical umbrella.

Key Insights

  • VAE as Non-linear PCA: In PCA, eigenvalues indicate the variance captured by a component. This paper shows that in a VAE, the term $(\beta/2) E[\sigma_j^{-2}]$ serves a nearly identical purpose, enabling the ranking of latent dimensions by quantitative importance.
  • The Meaning of $\beta$: Rather than a mere hyperparameter, $\beta$ represents the allowable distortion in the isometric space, setting the "resolution" at which the VAE represents data.
  • Density Estimation: Because the mapping is implicitly isometric, the Jacobian determinant becomes tractable. This allows for the calculation of the actual probability density of the input data $p(x)$ directly from the VAE’s internal parameters.

Experimental Results

The theory was validated across toy datasets, CelebA, MNIST, and several anomaly detection benchmarks.

Correlation between estimated and true generation probability (R=0.910).

In toy experiments, the proposed method for estimating data probability showed a high correlation (R ≈ 0.91) with the true distribution, significantly outperforming estimates based solely on the prior. Furthermore, by applying this understanding of $p(x)$ to anomaly detection, a standard "vanilla" VAE achieved state-of-the-art F1 scores (e.g., 0.9642 on KDDCUP99), outperforming specialized models like DAGMM and GMVAE.

Conclusion

By reframing the VAE as a non-linearly scaled isometric embedding, this work provides a rigorous quantitative foundation for understanding latent representations. This perspective not only demystifies the role of $\beta$ and posterior variance but also enables practical applications in feature ranking and high-accuracy density estimation for anomaly detection.

Summary updated

Abstract

Variational autoencoder (VAE) estimates the posterior parameters (mean and variance) of latent variables corresponding to each input data. While it is used for many tasks, the transparency of the model is still an underlying issue. This paper provides a quantitative understanding of VAE property through the differential geometric and information-theoretic interpretations of VAE. According to the Rate-distortion theory, the optimal transform coding is achieved by using an orthonormal transform with PCA basis where the transform space is isometric to the input. Considering the analogy of transform coding to VAE, we clarify theoretically and experimentally that VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter. As a result, we can estimate the data probabilities in the input space from the prior, loss metrics, and corresponding posterior parameters, and further, the quantitative importance of each latent variable can be evaluated like the eigenvalue of PCA.
BibTeX
@inproceedings{nakagawa2021quantitative,
  author = {Akira Nakagawa and Keizo Kato and Taiji Suzuki},
  title = {{Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=54NHILps4P},
}
Figure