In probabilistic machine learning, we often want to maximize the marginal likelihood of our data, $p(x)$. Because this is usually intractable, standard Variational Autoencoders (VAEs) optimize the Evidence Lower Bound (ELBO). However, the ELBO is often a loose bound, limited by the expressivity of the approximate posterior. A powerful family of methods has emerged to tighten this bound by using multiple samples and importance weighting.
The Core Mechanism: Averaging Before the Logarithm
The standard ELBO calculates the expectation of the log-likelihood ratio. In contrast, the importance-weighted objective—first introduced by Baker Grosse et al.—changes the order of operations: it averages multiple importance weights before taking the logarithm.
Mathematically, if we draw $K$ samples from a proposal distribution $q(z|x)$, the objective becomes $\mathbb{E}[\log \frac{1}{K} \sum_{i=1}^K \frac{p(x, z_i)}{q(z_i|x)}]$. By Jensen's inequality, increasing $K$ produces a strictly tighter lower bound that converges to the true log-likelihood as $K \to \infty$. As Baker Grosse et al. explain, this reduces the penalty on individual samples that poorly explain the data, allowing the model to explore a wider range of the latent space.
Why It Works: Expressivity and Augmentation
There are two primary ways to intuit why this multi-sample approach outperforms standard variational inference:
- Implicit Posterior Expressivity: Cremer et al. argue that the importance-weighted objective is actually a standard ELBO in disguise, but one that uses a much more powerful "implicit" posterior. By using $K$ samples, the model effectively creates a complex, multi-modal distribution that can capture structural details far beyond a simple Gaussian.
- Augmented Latent Spaces: Domke and Sheldon provide a different perspective, showing that importance-weighted inference is equivalent to standard variational inference performed on an augmented joint distribution space. They demonstrate that the looseness of the bound is directly proportional to the variance of the importance weights. Therefore, the goal of these objectives is essentially weight variance minimization.
Improving Stability and Accuracy
While increasing the number of samples $K$ improves the bound, it also introduces new challenges in gradient variance and computational cost.
- Ensemble Weighting: To create even more robust estimates, Kviman et al. propose using the "balance heuristic" from multiple importance sampling. By weighting samples against the average density of an ensemble of proposals rather than a single one, they achieve a tighter bound on the marginal log-likelihood.
- Sandwiching the Truth: While most work focuses on lower bounds, Struski et al. developed a way to use importance sampling to create a convergent upper bound. This allows practitioners to "sandwich" the true log-likelihood between two values, providing a precise measure of the variational gap.
- Handling Dependencies: Standard importance weighting assumes independent samples. Shu et al. extended this to statistically dependent samples generated during iterative refinement (like MCMC), allowing the model to benefit from the entire history of the inference process.
Applications Across Domains
The ability to trade increased computation for higher posterior accuracy has made importance-weighted objectives a "plug-and-play" improvement for diverse tasks:
- Reinforcement Learning: Tang et al. introduced JEPO, which uses multi-sample trajectories to optimize language models on "unverifiable" data like long-form reasoning, where traditional rewards are hard to define.
- Generative Modeling: The technique has been successfully applied to discrete latent variables via relaxations like DVAE# (Khoshaman and Amin) and to neural image compression, where Xu et al. showed it prevents posterior collapse and saves significant bitrate.
- Multimodal Fusion: In models dealing with multiple data types (like images and text), Siddharth N et al. and Shi et al. use importance weighting to ensure that one modality doesn't dominate the latent space, leading to more coherent joint representations.
- Gaussian Processes: For complex regression tasks, Salimbeni et al. and Haitao Liu 0002 et al. demonstrated that importance-weighted bounds reduce gradient variance and improve predictive performance in Deep and Multi-task Gaussian Processes.
By shifting the focus from single-point estimates to multi-sample averages, this research direction provides a principled way to scale the accuracy of probabilistic models alongside available computational power.
Key Foundational Work
Importance Weighted Autoencoders Yuri Burda (2016) — contributed 3 concept elements to this direction.
Importance Weighted Autoencoders (2016) — contributed 3 concept elements to this direction.
Reweighted Wake-Sleep (2015) — contributed 2 concept elements to this direction.