Abstract
Hierarchical models represent a challenging setting for inference algorithms. MCMC methods struggle to scale to large models with many local variables and observations, and variational inference (VI) may fail to provide accurate approximations due to the use of simple variational families. Some variational methods (e.g. importance weighted VI) integrate Monte Carlo methods to give better accuracy, but these tend to be unsuitable for hierarchical models, as they do not allow for subsampling and their performance tends to degrade for high dimensional models. We propose a new family of variational bounds for hierarchical models, based on the application of tightening methods (e.g. importance weighting) separately for each group of local random variables. We show that our approach naturally allows the use of subsampling to get unbiased gradients, and that it fully leverages the power of methods that build tighter lower bounds by applying them independently in lower dimensional spaces, leading to better results and more accurate posterior approximations than relevant baselines.
Introduction
Hierarchical models are fundamental tools in modern statistics, enabling the analysis of structured data across fields such as ecology, political science, and recommender systems. These models typically distinguish between "global" variables—representing shared characteristics across a population—and "local" variables—representing specific attributes of individual groups or users.
Despite their utility, performing accurate and scalable inference on these models remains a significant challenge. Researchers often face a trade-off between the speed of Variational Inference (VI) and the accuracy of Monte Carlo-based tightening methods. Standard VI is computationally efficient and supports data subsampling, but its reliance on simple distributions (e.g., Gaussians) often fails to capture the complexity of the true posterior. Conversely, tightening methods like Importance Weighted Variational Inference (IWVI) improve accuracy but suffer from the curse of dimensionality and typically require full-batch processing, making them impractical for large-scale datasets.
The Problem: The Global Tightening Bottleneck
The primary limitation of existing tightening methods in a hierarchical context is their global application. When importance weighting is applied to the entire model simultaneously, the variance of the weights increases exponentially with the dimensionality of the latent space. In hierarchical models, where the number of local variables grows with the number of groups, this dimensionality quickly becomes unmanageable. Furthermore, traditional importance-weighted bounds are not amenable to the stochastic optimization techniques (subsampling) that are essential for training on large-scale data.
Proposed Method: Locally Enhanced Bounds
To address these limitations, this research introduces locally-enhanced bounds. The core innovation lies in decomposing the evidence lower bound (ELBO) and applying "tightening operators" independently to each group of local variables.
Local Tightening Operators
Instead of attempting to tighten the bound for the entire model at once, the authors treat the tightening process as a modular operator. By applying this operator to each local group, they create a new family of variational objectives. The paper explores two primary implementations:
- Local Importance Weighting (Local IW): Adapts the logic of Importance Weighted Autoencoders (IWAE) to individual local groups.
- Local Uncorrected Hamiltonian Annealing (Local UHA): Utilizes a differentiable MCMC-inspired approach to refine local posterior approximations.
Scalability through Subsampling
Because the locally-enhanced objective remains a sum over independent groups, the method naturally supports stochastic gradients. This allows for "minibatch" training: one can sample a subset of groups, calculate the locally tightened bounds for those groups, and obtain an unbiased gradient to update both global and local parameters. This preserves the efficiency of standard VI while incorporating the accuracy of Monte Carlo methods.
Experimental Evaluation
The authors evaluated the proposed methods against standard VI and global tightening baselines using both synthetic and real-world datasets.
Synthetic Hierarchical Models
Tests were conducted on models with varying numbers of groups ($M$) and observations per group ($N$). The results demonstrated that as the problem size increases, local methods maintain their performance while global methods degrade. Local IW consistently achieved tighter bounds than both plain VI and Global IW.
MovieLens 100K
In a collaborative filtering task using the MovieLens 100K dataset, each user was treated as a local group. The objective was to approximate the posterior over user preferences and global movie characteristics. Local IW emerged as the superior method, providing more accurate posterior approximations in a fraction of the time required by global methods, which were hampered by the inability to subsample.
Key Insights
- Dimensionality Management: Tightening methods are most effective in low-dimensional spaces. By isolating local variables, the authors leverage Monte Carlo power where the "curse of dimensionality" is least restrictive.
- The Global-Local Gap: The paper provides a theoretical analysis showing that even with perfect local tightening, a gap remains in the bound determined by the quality of the global variational distribution. This highlights that while local enhancement is a major step forward, the choice of the global variational family remains a critical factor.
Conclusion and Future Work
Locally enhanced bounds offer a powerful path forward for hierarchical inference, combining the accuracy of multi-sample bounds with the scalability of stochastic optimization. While currently focused on two-level tree structures, future research may extend this framework to deeper hierarchies and more complex graphical models. Additionally, combining local enhancement with more flexible global approximations, such as Normalizing Flows, represents a promising avenue for further improving posterior accuracy.
Summary updated