Product-Form Doubly Reparameterized Gradients for High-Dimensional Latent Variable Models

Multi-sample variational methods train a latent variable model by averaging several importance weights before taking a logarithm. An importance weight is the ratio of how probable a sampled latent is under the model to how probable it is under the approximate posterior. The average gives a tighter bound on the data likelihood than the standard evidence lower bound (ELBO). For the encoder, the network that proposes latents, the gradient of this bound was once thought to get noisier with more samples. Doubly reparameterized gradients (DReG) fixed that: they make the encoder's signal-to-noise ratio improve with sample count. The repair only holds while the latent dimension stays fixed. When the latent space grows, the importance weights collapse onto a single sample, and every estimator falls back to the single-sample ELBO unless the sample count grows exponentially in dimension.

This note proposes a way out for the large class of models whose high dimension comes from many exchangeable groups: sets of images sharing a context, hierarchical Bayesian models, and grouped data with a shared global latent. The estimator averages weights block by block, within each group, instead of over the whole latent vector at once. That keeps the weight collapse local to each small block. DReG is then applied to this block-structured estimator. In two sentences: global importance weights degenerate exponentially in total latent dimension, but a product-form estimator only degenerates in the dimension of its largest block. Double reparameterization can be applied to that estimator, so the encoder keeps a high-signal gradient at scales where global DReG provably reduces to the ELBO.

Core Claim

Consider data split into groups $x_1,\dots,x_n$ with a global latent $c$ and local latents $z_j$. Assume the model factorizes as $p(c)\prod_j p(x_j, z_j\mid c)$ and the approximate posterior as $q_\phi(c\mid x)\prod_j q_\phi(z_j\mid x_j)$. Draw $K$ samples $c_k$ and, for every group, $K$ samples $z_{jl}$. Every global sample is then paired with every local sample, which gives the product-form estimate of the marginal likelihood:

$$\hat Z_{\text{PF}} = \frac{1}{K}\sum_{k=1}^K \frac{p(c_k)}{q_\phi(c_k\mid x)} \prod_{j=1}^n \frac{1}{K}\sum_{l=1}^K \frac{p(x_j, z_{jl}\mid c_k)}{q_\phi(z_{jl}\mid x_j)}.$$

Because the proposal factorizes across blocks, this estimate is unbiased for $p(x)$. $\mathbb{E}[\log \hat Z_{\text{PF}}]$ is therefore a valid lower bound by Jensen's inequality, and it costs $O(nK^2)$ likelihood evaluations. A global importance-weighted bound with $K^2$ joint samples has the same cost.

The estimator comes from Product-form estimators: exploiting independence to scale up Monte Carlo, which shows that exploiting independence turns exponential Monte Carlo error growth into linear growth. In their toy models it reaches 1% accuracy with $10^6$ samples where a standard estimator needs $10^{18}$, and it avoids weight degeneracy in a 100-dimensional hierarchical model. That paper never differentiates the estimator or trains an encoder with it. Overdispersed Black-Box Variational Inference supports the same intuition that local weights avoid global variance blow-up, but it works only with score-function gradients.

Our contribution is to derive and study a doubly reparameterized gradient for $\log \hat Z_{\text{PF}}$. We write $\log\hat Z_{\text{PF}}$ as a function $F$ of the individual log-weights. Each sampled variable $u$ (either a $c_k$ or a $z_{jl}$) then enters through a score term, $-s_u\,\nabla_\phi \log q_\phi(u)$, where $s_u = \partial F/\partial \log w_u$ is that sample's total responsibility. For a local sample, $s_u$ is its within-group normalized weight averaged over the global samples, weighted by their outer normalized weights. The DReG identity replaces each such term with a pathwise term:

$$\mathbb{E}\big[s_u\,\nabla_\phi \log q_\phi(u)\big] = \mathbb{E}\Big[\frac{\partial s_u}{\partial u}\frac{\partial u}{\partial \phi}\Big],$$

with the other samples held fixed. Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives shows that for ordinary importance-weighted autoencoders (IWAE) this identity yields the familiar squared-normalized-weight form. Applied here, it gives an unbiased estimator that can be implemented with stop-gradient operations. It has no raw score terms, and its responsibilities concentrate only as fast as each block's weights do.

The quantitative prediction follows from the high-dimensional collapse theorem of Learning with Importance Weighted Variational Inference: Asymptotics for Gradient Estimators of the VR-IWAE Bound. There, the log-weight variance grows linearly in dimension, so multi-sample gradient benefits require $\log K$ to grow like the dimension. Under the same assumption applied per block, we expect:

  • Local weights: the within-group weights stay healthy once $\log K \gtrsim d_z$.
  • Global weights: the global-latent weights stay healthy once $\log K \gtrsim d_c$.
  • Global DReG: the global estimator needs $\log K \gtrsim d_c + n d_z$.

The product-form DReG encoder SNR should therefore stay roughly flat as $n$ grows, while global DReG falls to ELBO level.

One limitation has to be stated at the outset. The outer weights on $c_k$ still multiply $n$ group-level estimates. If $q_\phi(c\mid x)$ is poorly matched to a posterior that sharpens with $n$, those outer weights will degenerate. The proposal moves the curse of dimensionality from the total latent dimension onto the global posterior's concentration. It does not remove it. Measuring how much of the problem this leaves is part of the program.

Research Program

The first part is derivation. We will write the product-form DReG estimator for the two-level model above and prove it is unbiased. We will then extend the SNR analysis of Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives and Tighter Variational Bounds are Not Necessarily Better to a joint limit where $n$ and $K$ grow together. The target result is an encoder SNR that depends on $n$ only through the outer-weight term.

When the prior over $c$ has learnable parameters, we will adopt the re-expression trick of Generalized Doubly Reparameterized Gradient Estimators. That work extends DReG to hierarchical models and trainable priors but always uses one global weight per joint sample. We reuse its method for the prior score terms and replace its global weighting with the block structure.

The second part is diagnostics. Following the protocol of Tighter Variational Bounds are Not Necessarily Better and On the difficulty of unbiased alpha divergence minimization, we will measure each estimator's gradient SNR, squared bias against exact gradients, and effective sample size at the block and outer levels.

The third part is training. We will train amortized models in which total latent dimension grows with the number of groups. We will then evaluate held-out likelihood with sandwich estimates in the style of Bounding Evidence and Estimating Log-Likelihood in VAE, which bound the evidence from both sides. This lets us report gaps rather than only lower bounds.

Baselines

Plain ELBO. The naive reference point is the standard single-sample ELBO trained with sticking-the-landing gradients, run at the same budget of $nK^2$ likelihood evaluations spent as extra independent single-sample estimates. On the difficulty of unbiased alpha divergence minimization shows that ordinary VI loses SNR only linearly in dimension, so the ELBO is a robust floor. The proposal must beat it on held-out log-likelihood and posterior mass coverage, not merely match its stability.

Global DReG. The strongest existing method is DReG on the global importance-weighted bound from Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives. Where a trainable prior is used, we run it in the generalized form of Generalized Doubly Reparameterized Gradient Estimators, which reports under 10% training-time overhead on MNIST, Omniglot and FashionMNIST. It is given $K^2$ joint samples so that its likelihood evaluations match ours exactly. It gets the same architectures, optimizer, learning-rate sweep and number of seeds. No reported numerical NLL result was found on Lacuna for DReG or GDREGs on these benchmarks, since the file gives only figures. We will therefore reproduce it ourselves. Beating it means a statistically clear improvement in sandwich-bounded held-out log-likelihood once $n d_z$ exceeds roughly $\log(K^2)$, and matching it below that threshold.

Tuned VR-IWAE. We also run the VR-IWAE bound with $\alpha$ tuned over a grid including 0.5. Alpha-divergence Variational Inference Meets Importance Weighted Auto-Encoders: Methodology and Asymptotics reports $\alpha=0.5$ as the best setting on MNIST, with no raw NLL value in the file.

Ablations isolating the key ingredient. The first ablation is the same product-form bound trained with ordinary reparameterized gradients. By the analysis of Tighter Variational Bounds are Not Necessarily Better, its within-group encoder SNR should shrink as $1/\sqrt{K}$, so this separates the benefit of the bound from the benefit of double reparameterization. The second ablation is a nested estimator that draws fresh local samples for each global sample. It is equally unbiased and has the same cost, and it isolates whether sharing local samples across global samples matters.

Why not just…?

Why not just use global DReG with more samples? Global DReG is the direct competitor, and at low total dimension it should tie. Learning with Importance Weighted Variational Inference: Asymptotics for Gradient Estimators of the VR-IWAE Bound proves that when $\log N = o(d)$ both reparameterized and doubly reparameterized SNR fall back to the single-sample value. Their linear Gaussian experiments show empirical SNR flatlining at $d=100$ and $d=500$. On the difficulty of unbiased alpha divergence minimization reports that a 128-dimensional Gaussian with $\alpha=0.4$ needs about $4\times10^9$ samples per step. The sweep over $n$ at matched compute tests this directly. If global DReG keeps pace as $n$ grows, the proposal fails.

Why not just train with the plain ELBO, since it only degrades linearly? This is a real threat for anyone who only cares about optimization stability. The case for a tighter bound rests on better likelihoods and less mode-seeking posteriors. Empirical Evaluation of Biased Methods for Alpha Divergence Minimization shows that in high dimensions, multi-sample methods silently revert to the exclusive-KL solution, which is exactly the ELBO answer. If the product-form bound, measured at convergence by sandwich estimates, is no tighter than the ELBO, it adds nothing. The hierarchical Gaussian experiment with exact posteriors settles this by comparing recovered posterior variances against the truth.

Why not just use a nested or semi-amortized estimator, IWAE within groups and ELBO on the global latent? Semi-amortized training is simpler and avoids the $K^2$ pairing. However, an ELBO on $c$ gives up mass coverage exactly where the posterior is shared across groups. The fresh-sample nested ablation and an ELBO-on-$c$ variant are both run. If either matches product-form DReG, we will report that the gain comes from block structure alone and that the shared-sample construction is unnecessary.

Why not just drop the score term (sticking the landing) or rely on Adam? Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives shows that sticking the landing is biased for any multi-sample bound, and the bias grows more complex once responsibilities couple across blocks. A product-form sticking-the-landing variant is cheap to include, so we measure its squared bias on the exact-gradient benchmark. On the difficulty of unbiased alpha divergence minimization shows that Adam's step shrinks with SNR and flatlines around $d\approx 40$, so the optimizer cannot substitute for a better estimator.

Proposed Experiments

Linear Gaussian hierarchy. The decisive check extends the linear Gaussian benchmark used by Tighter Variational Bounds are Not Necessarily Better and Learning with Importance Weighted Variational Inference: Asymptotics for Gradient Estimators of the VR-IWAE Bound. It becomes a two-level linear Gaussian hierarchy in which the marginal likelihood, the bound, and exact gradients are all available in closed form. We fix $d_c$ and $d_z$, sweep $n$ so that total dimension ranges from about 10 to 1000, and hold likelihood evaluations fixed. We then plot:

  • encoder SNR,
  • squared bias,
  • block-level and outer effective sample size,
  • bound gap.

The prediction is that global DReG SNR drops to ELBO level once $n d_z$ passes about $2\log K$, while product-form DReG stays near its small-$n$ value until outer-weight degeneracy sets in. We will also report where that outer failure begins as a function of how well $q(c)$ matches the posterior.

Kuntz et al. hierarchical model. Second, we train on the 100-dimensional Bayesian hierarchical model of Product-form estimators: exploiting independence to scale up Monte Carlo, with amortized encoders. Posterior marginals are compared against long MCMC runs.

Omniglot set-level model. Third, we fit a set-level generative model on Omniglot. Each set of images of one character shares a global context latent, and each image has its own latent. We vary set size to scale total dimension, and report sandwich-bounded held-out log-likelihood against all baselines above.

Negative control. Bayesian logistic regression on Sonar ($d=61$) and a1a ($d=120$), as used in Empirical Evaluation of Biased Methods for Alpha Divergence Minimization, has no block structure. There the method reduces to global DReG, and we expect no gain. Showing none would confirm that any improvements elsewhere come from factorization rather than from tuning.