Abstract
We propose a framework called HyperVAE for encoding distributions of distributions. When a target distribution is modeled by a VAE, its neural network parameters $\theta$ is drawn from a distribution $p(\theta)$ which is modeled by a hyper-level VAE. We propose a variational inference using Gaussian mixture models to implicitly encode the parameters $\theta$ into a low dimensional Gaussian distribution. Given a target distribution, we predict the posterior distribution of the latent code, then use a matrix-network decoder to generate a posterior distribution $q(\theta)$. HyperVAE can encode the parameters $\theta$ in full in contrast to common hyper-networks practices, which generate only the scale and bias vectors as target-network parameters. Thus HyperVAE preserves much more information about the model for each task in the latent space. We discuss HyperVAE using the minimum description length (MDL) principle and show that it helps HyperVAE to generalize. We evaluate HyperVAE in density estimation tasks, outlier detection and discovery of novel design classes, demonstrating its efficacy.
Introduction
In meta-learning, the objective is to develop models that can rapidly adapt to new tasks. While supervised approaches like MAML are effective, they require labeled data for adaptation. In unsupervised settings, where labels are unavailable, capturing the underlying structure of task distributions becomes challenging. Existing "hyper-networks"—models that generate weights for other networks—often struggle with the high dimensionality of neural network parameters. Consequently, most hyper-networks are restricted to generating simple scaling or bias factors, which limits their ability to capture the full complexity of specific data distributions.
HyperVAE addresses these challenges by learning a manifold of models rather than just data points. By generating full parameter sets within a hierarchical generative framework, HyperVAE preserves task-specific information in a searchable latent space, enabling unsupervised meta-learning and robust outlier detection.
Methods
HyperVAE is structured as a hierarchical Variational Autoencoder. At the primary level, a standard VAE models data points $x$ using a latent variable $z$. At the "hyper" level, a second VAE models the distribution of the primary VAE's parameters $\theta$ using a task-level latent variable $u$.
Matrix Hyper-Decoder
To manage the high dimensionality of neural network weights, HyperVAE employs a matrix generation network. Instead of standard fully-connected layers, weights are treated as structured matrices: $W = \sigma(UHV + B)$. This approach reduces the parameter count for weight generation by three orders of magnitude (e.g., from 64 million to 176 thousand), making full parameter generation feasible.
Minimum Description Length (MDL) Objective
The training objective is derived from the MDL principle, which frames learning as a data compression problem. The optimal model minimizes the combined description length of the model parameters and the data misfits. This provides a principled information-theoretic regularization that prevents the hyper-network from over-complexifying, ensuring a compact latent representation of task knowledge.
Results
The model was evaluated on MNIST, Omniglot, Fashion MNIST, and a specialized Aluminum Alloys dataset.
Density Estimation
HyperVAE consistently achieved lower Negative Log-Likelihood (NLL) than standard VAEs and MetaVAEs across multiple datasets.
On MNIST, HyperVAE reached an NLL of 88.2, significantly outperforming the standard VAE (99.4) and MetaVAE (93.0). Similar improvements were observed on Omniglot and Fashion MNIST, demonstrating superior density estimation capabilities.
Outlier Detection and Novelty Discovery
HyperVAE demonstrated high robustness in identifying unseen data. On MNIST, it achieved an AUC of 95.3% for outlier detection, compared to 93.0% for a standard VAE and 54.7% for MetaVAE. In the alloy design experiment, the framework successfully identified new alloy compositions close to target properties by navigating the latent $u$-space (tasks) and $z$-space (data) using Bayesian Optimization.
Discussion and Limitations
While HyperVAE offers significant advantages in capturing task distributions, it introduces computational overhead. The nested architecture roughly doubles the algorithmic complexity compared to a standard VAE. Additionally, generating weights on-the-fly for every task can increase latency, though parallelization via batched matrix multiplication may mitigate this. Future work includes exploring 3D tensor-based weight generation and applying the framework to more complex design spaces.
Summary updated