A variational autoencoder (VAE) is a generative model that compresses each input into a small random code and learns to rebuild the input from that code. It has a well-known failure mode called posterior collapse. When the decoder is powerful, for example an LSTM, Transformer or PixelCNN that can predict the next token or pixel from what it has already generated, it learns to ignore the code entirely. The code then carries zero bits about the input. Today the only way to know whether a given encoder–decoder architecture will collapse is to train it and watch the KL term and the count of active latent units fall to zero. That makes searching over where and how the latent code is wired into the decoder expensive. It also leaves unresolved whether collapse fixes built for one backbone survive on another.
We propose a training-free score, computed from two backward passes on one minibatch at initialization, that predicts whether an architecture will collapse. The score measures what fraction of the decoder's initial learning signal depends on the correct latent code being paired with the input, as opposed to a code drawn at random. The intuition comes from the "two-path" account of collapse: a latent path and a local-context path race, and the one that learns faster early wins. If the race can be read off the first gradient, a search can reject collapse-prone architectures in seconds. The same number can then be tested for whether it keeps its meaning across LSTM, Transformer and convolutional decoders.
Core Claim
Posterior collapse is largely decided by an early learning-rate race. That race is visible at initialization as the share of the parameter gradient that depends on the input–code pairing. Let $\ell(\theta; x, z, c)$ be the reconstruction loss of a decoder with parameters $\theta$ on input $x$, given latent code $z \sim q_\phi(z|x)$ and autoregressive context $c$. Let $\tilde z$ be a code taken from a different example in the same batch, which is a sample from the aggregate posterior with the pairing to $x$ broken. We define the pairing gradient share
$$S = \frac{\mathbb{E}\,\lVert \nabla_\theta \ell(\theta; x, z, c) - \nabla_\theta \ell(\theta; x, \tilde z, c) \rVert^2}{\mathbb{E}\,\lVert \nabla_\theta \ell(\theta; x, z, c) \rVert^2},$$
where expectations are over a single minibatch. $S$ near zero means the decoder's first updates would look the same whether or not it saw the right code. In that case the context path will win and the KL penalty will push the posterior onto the prior. The claim is that $S$ at initialization rank-correlates with final mutual information and active units across a heterogeneous space of decoder backbones and latent-injection schemes. We further claim it does so better than existing zero-cost proxies, better than decoder capacity counts, and as well as short partial training at a fraction of the cost.
Grounding in Prior Work
The mechanism rests on Cyclical Annealing. That paper frames collapse as competition between a latent path and a local-history path, and fixes it by repeatedly resetting the KL weight. We keep its diagnosis but try to measure the race rather than reset it. Bowman et al. showed that collapse appears exactly when decoders are expressive, and SKIP-VAE showed that deeper decoders collapse more and that feeding $z$ into every layer prevents this in CNNs and LSTMs. These two papers motivate treating latent wiring as an architecture variable. Neither searches over that wiring, and neither tests self-attention decoders.
The information-theoretic side supplies both motivation and constraints. InfoVAE and McCarthy et al. show that the ELBO implicitly penalizes $I(X;Z)$, and they add an explicit mutual-information bonus. Forget-me-not lower-bounds $I(X;Z)$ with a contrastive critic that tells matched $(x,z)$ pairs apart from shuffled ones. Rezaabad trains a dual-form estimator alongside the model. All of these are training-time regularizers that require full runs. Our score borrows Forget-me-not's matched-versus-shuffled contrast but applies it to gradients at initialization, as a diagnostic rather than a loss term.
Wang et al. prove that collapse is equivalent to latent non-identifiability. Kinoshita et al. show that an inverse-Lipschitz decoder lower-bounds the posterior's divergence from the prior. Both enforce these properties with gradients of input-convex networks, which costs second-order derivatives and rules out standard backbones. We treat their results as a constraint on our proxy. Static decoder sensitivity to $z$ is the quantity their theory controls, so our ablation compares $S$ against a cheap Jacobian estimate of that sensitivity. If static sensitivity alone predicts collapse, the gradient-race framing is unnecessary.
On the search side, RMI-NAS scores architectures with a single-batch dependence measure (normalized HSIC) against a reference network, but only for discriminative classifiers. EZNAS evolves zero-cost proxies that transfer across vision search spaces. LPZero shows that such vision proxies collapse to near-chance rank correlation on Transformers. That last finding is the main threat to any claim that a proxy generalizes across architectures. We address it directly by testing calibration across backbone families rather than assuming it. PGNAS uses "posterior" for a Bayesian posterior over architectures. It is unrelated to latent collapse, and we cite it only to mark that distinction.
Research Program
The first deliverable is a benchmark, because none exists. We define a search space over the latent interface. Decoder backbone ranges over LSTM, causal Transformer and PixelCNN-style convolution. The latent injection site ranges over first-input-only, every-layer skip in the manner of SKIP-VAE, cross-attention to a projected code, and a prefix token. The space also varies decoder depth, width, latent dimension, and encoder-to-decoder capacity ratio. We sample several hundred configurations and train each to convergence on Yahoo and Yelp (text, Transformer and LSTM decoders) and on Omniglot and Fashion-MNIST (images, convolutional decoders). For each configuration we record final mutual information, active units, NLL and KL. Every configuration is trained under the plain ELBO with a fixed monotonic warm-up, so that architecture is the only variable. A smaller subset is also trained under cyclical annealing, to check whether the score's predictions hold under a different schedule.
The second deliverable is the score itself, together with its analysis. We compute $S$ with one minibatch at initialization, at a cost of two backward passes. We then test how it tracks the actual race. For a subset of runs, we log the per-step gradient share throughout training and check whether the share at step zero predicts the step at which KL first falls below a threshold. We also derive a cheap variant that restricts the gradient to decoder parameters only, and one that adds the KL gradient on the encoder. This tells us whether the prior's pull must be included for the score to work.
The third deliverable is a search. We use the score inside regularized evolution over the interface space, pick the top architectures, and train them fully. This checks whether ranking power holds in the top decile, where EZNAS reports that zero-cost proxies degrade. As a side product, the benchmark directly answers the open question of whether every-layer skip injection alone prevents collapse in Transformer decoders. It also shows whether cross-attention is needed, as the heavier recipe in McCarthy et al. might suggest.
Baselines
The naive reference point is parameter count, together with the decoder-to-latent capacity ratio. LPZero reports that on Transformers these trivial counts beat several established proxies. If our score cannot beat "smaller decoders collapse less", it has no value. Both are computed on the same configurations, and beating them means a statistically higher Kendall $\tau$ against final active units, with bootstrap confidence intervals over configurations.
The strongest existing zero-cost competitors are EZNAS-A, SNIP, Synflow and an adapted RMI-NAS score. For RMI-NAS, HSIC is computed between $z$ and decoder outputs rather than against a teacher. All are run on the identical minibatch and seed. For reference, the only numbers reported for these proxies are on other tasks. EZNAS-A reaches Kendall $\tau = 0.65$ on NAS-Bench-201 CIFAR-10 (EZNAS). LPZero reaches Spearman $\rho = 0.748$ on FlexiBERT while SNIP gets 0.173 (LPZero). No reported result was found on Lacuna for any proxy predicting posterior collapse, so the comparison happens entirely on our benchmark. We will re-evolve an EZNAS-style proxy on our own training split, so that the learned-proxy baseline gets the same data we do.
The most dangerous baseline is budgeted partial training, following Budgeted Performance Estimation. Each configuration is trained for $k$ steps and ranked by its KL or active units at step $k$. The budget $k$ is swept from equal wall-clock cost with our score up to 5% of full training. Beating it means matching its rank correlation at a lower compute budget. It does not mean exceeding it at every budget.
For the downstream search, the reference methods are the hand-designed SKIP-VAE decoder and the Forget-me-not hybrid critic. On Yahoo, Forget-me-not reports 2.8 MI and 11.0 active units where a standard VAE got 0.0 and 0.0 (Forget-me-not). SKIP-SA-VAE keeps all 64 latent dimensions active (SKIP-VAE). Architectures found by our search must match these on MI and active units without an auxiliary critic, and at equal or better NLL. Search compute plus final training must be counted against their training compute.
The key ablation replaces $S$ with the static sensitivity term alone. This is an estimate of the smallest singular value of the decoder Jacobian with respect to $z$, which is the quantity inverse-Lipschitz theory in Kinoshita et al. controls. The ablation isolates whether pairing-dependent gradient flow, rather than mere sensitivity, carries the signal.
Why not just…?
Why not just put skip connections into every decoder and skip the search? This is plausible, and the benchmark tests it directly. If every-layer skips keep latents active across all backbones, including Transformers, then architecture selection for collapse is moot. We would report that as the main finding and scale the proxy's claims down to capacity choices. The evidence for skips comes only from CNNs and LSTMs (SKIP-VAE). Meanwhile, the Transformer work in McCarthy et al. needed discrete latents plus an auxiliary bag-of-words loss, which hints that attention may route around injected codes. We do not know which way this goes, and that uncertainty is why the experiment is worth running.
Why not just use cyclical annealing or a mutual-information bonus on any architecture? These change the optimization, not the architecture, and they add a hyperparameter per model: $\lambda$ and $\alpha$ in InfoVAE, or cycle count in Cyclical Annealing. Forget-me-not's standalone critic is reported to inflate mutual information at the expense of NLL. The two approaches are complementary rather than competing. The cyclical-annealing subset checks whether $S$ still ranks configurations under a collapse-mitigating schedule. If annealing makes all configurations equally healthy, the proxy matters less, and we will say so.
Why not just train for a few hundred steps and look at the KL? This is the real threat. Early KL is a direct readout of the race that $S$ tries to predict. If partial training at a small budget ranks configurations as well as $S$ does, our contribution reduces to a constant-factor speedup. The budget sweep in the baselines is designed to show exactly where the crossover lies. We commit to reporting the crossover even if it is at very small $k$.
Proposed Experiments and Decisive Checks
The first decisive check is rank correlation between $S$ and final active units and mutual information, pooled across backbones and also within each backbone. The pooled number is what matters for architectural generalization. A proxy that ranks well within LSTMs but assigns Transformers a systematically shifted scale would repeat the failure LPZero documents. We therefore also test whether a single threshold on $S$, chosen on LSTM configurations, classifies collapse on Transformer and convolutional configurations with held-out accuracy well above the base rate.
The second check is the top-decile test. Among the 10% of configurations that do not collapse, does $S$ still order them by final mutual information and NLL?
The third check is the mechanism test. Across training, does the step-zero pairing share predict when KL vanishes, and does the static-sensitivity ablation fail where $S$ succeeds? If the ablation matches $S$, the gradient-race story is wrong and we will report sensitivity as the useful quantity.
The final check is the end-to-end comparison on Yahoo, Yelp, Omniglot and Fashion-MNIST. Architectures chosen by $S$-guided search are compared against SKIP-VAE, Forget-me-not and a randomly sampled architecture. For Fashion-MNIST, the external reference is the NLL of 234.4 reported for IL-LIDMVAE against 258.8 for a standard VAE (Kinoshita et al.). All methods are compared at matched total GPU hours.
References
- Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
- Generating Sentences from a Continuous Space
- Avoiding Latent Variable Collapse with Generative Skip Models.
- InfoVAE: Balancing Learning and Inference in Variational Autoencoders.
- Addressing Posterior Collapse with Mutual Information for Improved Variational Neural Machine Translation
- Forget-me-not! Contrastive critics for mitigating posterior collapse
- Learning Representations by Maximizing Mutual Information in Variational Autoencoders
- Posterior Collapse and Latent Variable Non-identifiability
- Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network
- Neural Architecture Search with Representation Mutual Information
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- LPZero: Language Model Zero-cost Proxy Search from Zero
- Posterior-Guided Neural Architecture Search
- Rethinking Performance Estimation in Neural Architecture Search