Abstract
Out-of-distribution (OOD) detection is a critical task in machine learning, particularly for safety-critical applications where unexpected inputs must be reliably flagged. While hierarchical variational autoencoders (HVAEs) offer improved representational capacity over traditional VAEs, their performance is highly sensitive to how latent dimensions are distributed across layers. Existing approaches often allocate latent capacity arbitrarily, leading to ineffective representations or posterior collapse. In this work, we introduce a theoretically grounded framework for optimizing latent dimension allocation in HVAEs, drawing on principles from information theory to formalize the trade-off between information loss and representational attenuation. We prove the existence of an optimal allocation ratio $r^{\ast}$ under a fixed latent budget, and empirically show that tuning this ratio consistently improves OOD detection performance across datasets and architectures. Our approach outperforms baseline HVAE configurations and provides practical guidance for principled latent structure design, leading to more robust OOD detection with deep generative models.
Introduction
Deep generative models, specifically Variational Autoencoders (VAEs), have become a cornerstone for out-of-distribution (OOD) detection. Hierarchical VAEs (HVAEs) extend this capability by capturing data features at multiple levels of abstraction, from global structures to fine-grained textures. However, a significant challenge remains: how should latent dimensions be distributed across the hierarchy? Current practices often rely on arbitrary allocations, which can lead to either significant information loss in shallow layers or "posterior collapse" (attenuation) in deeper ones. This paper proposes a principled framework to optimize this allocation using a geometric progression ratio $r$, ensuring a balance between compression and information retention.
Methods: A Principled Approach to Latent Allocation
The core of our framework is the systematic distribution of a fixed total latent budget $b$ across $N$ layers. We define a latent dimension ratio $r$, where the dimensionality of adjacent layers follows a geometric progression: $r = l_{i+1} / l_i$. This allows the dimensionality of any layer $l_i$ to be determined by a closed-form formula.
Drawing on the Information Bottleneck (IB) principle, we formalize the trade-off between two extremes:
- Information Loss: Occurs when the bottleneck is too tight ($|Z| \to 0$), preventing the model from capturing essential data structures.
- Attenuation: Occurs when the bottleneck is too wide ($|Z| \to \infty$), causing the model to overfit to noise and making deeper latent layers semantically inefficient.
Using the Extreme Value Theorem, we mathematically prove that an optimal ratio $r^*$ exists that maximizes the model's utility for OOD detection. We validate this by estimating the Mutual Information $I(X; Z_i)$ between the input and various latent levels to observe how $r$ influences information flow.
Experimental Results
We evaluated our framework on several datasets, including FashionMNIST, MNIST, and CIFAR10, testing against various OOD datasets like SVHN and CelebA.
Key findings include:
- Dataset-Specific Optima: The "Goldilocks" ratio varies by dataset complexity. For instance, CIFAR10 performed best with a low ratio ($r=0.25$, aggressive compression), while FashionMNIST required a higher ratio ($r=0.75$) for optimal performance.
- Robustness of $r^*$: Once an optimal $r^*$ is identified for an in-distribution dataset, it remains optimal across various OOD test sets, providing a stable design guideline.
- Superiority over Baselines: Optimized $r^*$ configurations consistently outperformed standard HVAE baselines. In CIFAR10, the $r=0.25$ configuration (170 → 43 → 11 dimensions) significantly reduced the False Positive Rate (FPR95) compared to the $r=0.5$ baseline commonly used in literature.
Conclusion
This work demonstrates that the architectural design of HVAEs, specifically the allocation of latent dimensions, is a critical factor in OOD detection performance. By adopting a geometric progression ratio $r$, researchers can move away from heuristic-based design toward a theoretically grounded approach. Future work will explore dynamic ratios that vary between layers and the application of this framework to non-Gaussian latent distributions.
Summary updated