Variational Inference with Locally Enhanced Bounds for Hierarchical Models

Tomas Geffner, Justin Domke

Overview

Hierarchical models are foundational in modern statistics, powering applications from ecological modeling to personalized recommendation systems. These models typically rely on a combination of global variables (e.g., average movie preferences across all users) and local variables (e.g., a specific user's unique tastes).

However, performing inference—calculating the posterior distribution of these variables given data—presents a significant challenge. This paper introduces locally-enhanced bounds, a novel framework that makes Variational Inference (VI) for hierarchical models both more accurate and highly scalable. By applying "tightening" techniques locally to individual groups of variables rather than globally, the authors achieve superior posterior approximations while preserving the ability to process massive datasets efficiently.

The Inference Dilemma

Practitioners have historically faced a difficult trade-off when performing inference on hierarchical models:

  1. Markov Chain Monte Carlo (MCMC): Highly accurate, but scales poorly because it generally requires evaluating the entire dataset at every step.
  2. Variational Inference (VI): Fast and supports data subsampling (e.g., evaluating a few users at a time), but often relies on overly simplistic assumptions that lead to inaccurate posterior approximations.
  3. Global Tightening Methods: Techniques like Importance Weighted VI (IWVI) were designed to fix VI's accuracy issues. Unfortunately, they break down when combined with data subsampling. Furthermore, their performance degrades rapidly as the number of variables grows—a classic case of the "curse of dimensionality."

A "Best of Both Worlds" Approach

To resolve this dilemma, the authors propose shifting from a global to a local perspective. Instead of applying a tightening method to the entire model simultaneously (Global Importance Weighting), they apply it locally to each independent group of local variables.

Global weighting creates copies of every variable, whereas local weighting applies the technique independently to each group, preserving the model's natural structure.

This approach relies on a few key technical innovations:

  • Bounding Operators: A generalized mathematical framework to describe methods that provide a lower bound on the log-marginal likelihood (such as plain VI, Importance Weighting, or Hamiltonian Annealing).
  • Locally-Enhanced Bounds: A new objective function that utilizes a standard VI bound for global variables but substitutes more powerful, "tightened" bounds for each local group.
  • Subsampling Compatibility: Because the new objective is a sum of independent local terms, practitioners can randomly sample a subset of groups to get an unbiased estimate of the gradient. This restores the ability to train efficiently on large datasets.

The paper specifically evaluates two variants of this framework: Local IW (Importance Weighting) and Local UHA (Uncorrected Hamiltonian Annealing).

Key Insights

  • Beating the Curse of Dimensionality: Tightening methods struggle in high-dimensional spaces. In a hierarchical model with 1,000 groups, the total dimensionality is massive. However, by working locally, the tightening methods operate in a low-dimensional space (just the variables for one group), making them vastly more effective.
  • Structural Preservation: Standard global tightening methods often destroy the conditional independence inherent in hierarchical models. Locally-enhanced bounds explicitly leverage and preserve this natural structure.

Experimental Results

The researchers evaluated their method on both synthetic datasets (varying the number of groups $M$ and observations per group $N$) and real-world data (the MovieLens 100K benchmark). Across all experiments, locally-enhanced bounds consistently outperformed both plain VI and global tightening baselines.

Final lower bounds achieved on synthetic datasets. Local methods (yellow and blue lines) consistently reach higher bounds than plain VI (grey dashed) and global counterparts (green and red), especially as the number of groups increases.
  • Superior Accuracy: Local IW and Local UHA achieved significantly tighter bounds and better test log-likelihoods than plain VI.
  • Robust Scalability: As the number of groups ($M$) increased, the performance gap between local and global methods widened. Global methods became increasingly computationally expensive and less accurate, while local methods maintained high efficiency and performance.
  • Data Imbalance: The benefits of the local approach held true even in unbalanced scenarios where some groups contained significantly more observations than others.

Limitations and Future Directions

While highly effective, the current framework focuses primarily on two-level tree structures (one global level, one local level). Real-world models often feature deeper hierarchies (e.g., students within classrooms within schools), which will require extending this methodology. Additionally, while local variables are handled with powerful tightening tools, the global variables still rely on a standard variational family. If the global posterior is highly complex, a performance gap remains that local tightening alone cannot close. Future work may also explore algorithms that automatically detect conditional independence in arbitrary graphical models to construct these bounds without manual derivation.

Summary updated

Abstract

Hierarchical models represent a challenging setting for inference algorithms. MCMC methods struggle to scale to large models with many local variables and observations, and variational inference (V...
BibTeX
@inproceedings{geffner2022variational,
  author = {Tomas Geffner and Justin Domke},
  title = {{Variational Inference with Locally Enhanced Bounds for Hierarchical Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=iIIZFhuU91_},
}
Figure