A Filtering Approach to Stochastic Variational Inference

Neil Houlsby, David M. BleiNeurIPS 2014 · OpenReview

Overview

This paper reimagines Stochastic Variational Inference (SVI) not just as an optimization problem, but as a tracking problem. By utilizing a Kalman filter to "track" the optimal variational parameters through noisy data samples, the authors introduce an algorithm that automatically tunes its own learning rate and remains robust to heavy-tailed noise.

The Problem: The Nuisance of Step Sizes

Bayesian inference is the gold standard for reasoning under uncertainty, but it is notoriously difficult to scale to massive datasets. Stochastic Variational Inference (SVI) addressed this by using mini-batches of data to calculate noisy gradients, allowing models to process millions of documents or users efficiently.

However, SVI suffers from a "nuisance" common to all stochastic optimization: the step-size schedule (or learning rate). If the step size is too large, the algorithm bounces around erratically due to noise; if it is too small, convergence takes an impractically long time. Most researchers rely on hand-tuned, heuristic schedules (like the Robbins-Monro schedule) that fail to adapt to the specific contours and "wiggles" of the data. The authors aim to replace these hand-crafted schedules with a principled, probabilistic framework.

Methods: Variational Filtering

The authors introduce Variational Filtering (VF). Their key insight is to view SVI as an attempt to reach the "true" coordinate optimum—what you would get if you looked at all the data at once—using only a noisy estimate from a mini-batch.

  1. The Meta-Model: They define a "meta-model" where the optimal parameter is a hidden state, and each mini-batch provides a noisy observation of that state.
  2. Kalman Filtering: They use a Kalman filter to track this hidden state. In this setup, the Kalman gain (a standard term in control theory) becomes mathematically equivalent to the SVI step size.
  3. Gaussian vs. Student’s t:
  4. GVF (Gaussian Variational Filter): Assumes noise is Gaussian. This recovers standard SVI but with an automated step size.
  5. TVF (Student’s t Variational Filter): This is the paper's "secret sauce." Because stochastic gradients are often "spiky" or heavy-tailed, they use a Student’s t-distribution. This makes the algorithm robust to outliers; if a mini-batch produces a wild, atypical gradient, the filter recognizes it as an outlier and downweights its influence.
  6. Online Parameter Estimation: The filter estimates the noise levels and the signal on the fly using moving averages, removing the need for manual user tuning.
Comparison of tracking error and ELBO for Variational Filtering, SVI, and Constant Rate

Key Insights

  • Bayes' Rule as an Optimizer: Instead of picking a learning rate based on a rule of thumb, the algorithm uses Bayes' rule to determine how much to trust the current mini-batch.
  • The "Aha" Moment on Step Sizes: Standard SVI schedules are essentially a special, restricted case of a Kalman filter where the "process noise" is set to zero. By allowing this noise to be non-zero, the algorithm can adapt to non-stationary data.
  • Robustness to Spiky Gradients: The Student’s t-filter is significantly more stable than the Gaussian version, preventing the model from being knocked off course by a single anomalous mini-batch.
  • Automatic Bias-Variance Trade-off: The filter naturally balances bias (staying near the previous estimate) and variance (moving toward the new, noisy data point) without manual intervention.

Key Results

The authors evaluated their approach on two distinct tasks: Topic Modeling (LDA) using massive text corpora (ArXiv, NYT, Wikipedia), and Binary Matrix Factorization (BMF) for recommendation systems (Kosarak, BMS-WebView-2, Netflix prize).

  • Outperforming the "Oracle": The Student’s t-filter (TVF) performed as well as, or better than, an "oracle" constant learning rate chosen in retrospect after exhaustive tuning.
  • Superior to State-of-the-Art: In both Topic Modeling and Matrix Factorization, TVF outperformed the original SVI algorithm and other modern adaptive learning rate methods.
  • Faster Convergence: On the ArXiv dataset, TVF reached a high-quality solution much faster than the standard SVI schedule.
  • Handling Regime Changes: In experiments with non-stationary data—where the topic of documents changed every 5,000 samples—the Variational Filter reacted much more quickly to the shift than traditional methods.
Filter response to regime changes every 5000 documents

Limitations and Future Work

While highly effective, the approach has a few limitations. To maintain computational efficiency ($O(N)$), the authors used a "diagonal" approximation, treating each parameter dimension independently. A full-rank version would be $O(N^3)$, which is too slow for high-dimensional models. Additionally, because the posterior of a Student’s t-distribution isn't strictly closed-form when degrees of freedom are involved, the authors relied on a moment-matching approximation. Finally, the paper focuses on "global" parameters; applying this filtering approach to "local" parameters (those specific to a single data point) remains an open area for future exploration.

Summary updated

Abstract

Stochastic variational inference (SVI) uses stochastic optimization to scale up Bayesian computation to massive data. We present an alternative perspective on SVI as approximate parallel coordinate ascent. SVI trades-off bias and variance to step close to the unknown true coordinate optimum given by batch variational Bayes (VB). We define a model to automate this process. The model infers the location of the next VB optimum from a sequence of noisy realizations. As a consequence of this construction, we update the variational parameters using Bayes rule, rather than a hand-crafted optimization schedule. When our model is a Kalman filter this procedure can recover the original SVI algorithm and SVI with adaptive steps. We may also encode additional assumptions in the model, such as heavy-tailed noise. By doing so, our algorithm outperforms the original SVI schedule and a state-of-the-art adaptive SVI algorithm in two diverse domains.
BibTeX
@inproceedings{houlsby2014filtering,
  author = {Neil Houlsby and David M. Blei},
  title = {{A Filtering Approach to Stochastic Variational Inference}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=BkZiJK-O-r},
}
Figure