Abstract
In the Variational Autoencoder (VAE), the variational posterior often aligns closely with the prior, which is known as posterior collapse and hinders the quality of representation learning. To mitigate this problem, an adjustable hyperparameter beta has been introduced in the VAE. This paper presents a closed-form expression to assess the relationship between the beta in VAE, the dataset size, the posterior collapse, and the rate-distortion curve by analyzing a minimal VAE in a high-dimensional limit. These results clarify that a long plateau in the generalization error emerges with a relatively larger beta. As the beta increases, the length of the plateau extends and then becomes infinite beyond a certain beta threshold. This implies that the choice of beta, unlike the usual regularization parameters, can induce posterior collapse regardless of the dataset size. Thus, beta is a risky parameter that requires careful tuning. Furthermore, considering the dataset-size dependence on the rate-distortion curve, a relatively large dataset is required to obtain a rate-distortion curve with high rates. Extensive numerical experiments support our analysis.
Introduction
Variational Autoencoders (VAEs) are a cornerstone of generative modeling, yet they are plagued by a failure mode known as posterior collapse. This occurs when the model's latent variables fail to capture meaningful information, effectively reverting to the prior distribution. While practitioners use the $\beta$-VAE framework to balance reconstruction quality (distortion) and latent information (rate), the theoretical boundary of this collapse has been unclear. This paper provides a precise mapping of the "danger zone" where posterior collapse becomes inevitable, regardless of dataset size.
Theoretical Framework: The Linear VAE
To derive a sharp theoretical understanding, the researchers analyze a Linear VAE in the high-dimensional limit. In this regime, the number of samples $n$ and the dimensionality $d$ both go to infinity, while their ratio $\alpha = n/d$ (sample complexity) remains constant. By applying the Replica Method from statistical physics, the authors derived asymptotic formulas for signal recovery and the Rate-Distortion (RD) curve.
The model is tested against the Spiked Covariance Model (SCM), a synthetic benchmark where a clear signal is buried in noise. This allows for exact comparisons between theoretical predictions and experimental results, which are further validated on real-world datasets like MNIST and CIFAR-10.
Key Findings: The Threshold of Collapse
The analysis reveals that $\beta$ is a "risky" parameter. Unlike standard regularization, if $\beta$ exceeds a specific threshold related to signal strength and noise, the model will collapse regardless of how much data is provided.
As shown in the quantitative results, a "plateau" in generalization error emerges when $\beta$ is large. In this region, the model fails to learn until a critical amount of data (sample complexity $\alpha$) is reached. If $\beta$ is set too high (e.g., $\beta_{VAE} = 2.0$ in the figure), the generalization error remains at its maximum and the rate (KL divergence) stays at zero indefinitely, signifying total posterior collapse.
Data-Hungry RD Curves
The research also demonstrates that the Rate-Distortion curve is highly sensitive to dataset size. To achieve "high-rate" representations—those that capture complex, detailed features—the model requires a much larger $\alpha$ than is needed for simple, low-rate reconstructions. This suggests that achieving high-quality representation learning is more data-intensive than previously assumed.
Conclusion
This work provides a rigorous theoretical foundation for tuning $\beta$-VAEs. It establishes that posterior collapse is not just a result of insufficient data, but can be fundamentally induced by the regularization parameter itself. The derived phase diagrams and closed-form expressions offer practical guidelines for avoiding the collapse regime and estimating the data requirements for effective representation learning.
Summary updated