Variational Information Planning for Sequential Decision Making

Jason Pacheco, John W. Fisher IiiAISTATS 2019 · OpenReview

Abstract

We consider the setting of sequential decision making where, at each stage, potential actions are evaluated based on expected reduction in posterior uncertainty, given by mutual information (MI). As MI typically lacks a closed form, we propose an approach which maintains variational approximations of, both, the posterior and MI utility. Our planning objective extends an established variational bound on MI to the setting of sequential planning. The result, variational information planning (VIP), is an efficient method for sequential decision making. We further establish convexity of the variational planning objective and, under conditional exponential family approximations, we show that the optimal MI bound arises from a relaxation of the well-known exponential family moment matching property. We demonstrate VIP for sensor selection, experiment design, and active learning, where it meets or exceeds methods requiring more computation, or those specialized to the task.

Introduction

In Bayesian sequential decision-making—whether a robot is choosing which sensor to activate or a scientist is selecting the next experiment—the goal is typically to maximize Mutual Information (MI). Maximizing MI identifies the action that most effectively reduces uncertainty about a hidden state. However, MI is notoriously difficult to compute, as it requires evaluating expectations over both the current posterior and the future posterior predictive distribution. Traditional approaches, such as Markov chain Monte Carlo (MCMC), are often too computationally intensive for real-time applications, while simpler "plug-in" estimators can be biased and require large sample sizes for accuracy.

Variational Information Planning (VIP)

To address these challenges, we introduce Variational Information Planning (VIP). The core of this framework is the maintenance of variational approximations for both the posterior distribution and the MI utility itself.

Key technical components include:

  • Variational MI Bound: We extend the Barber-Agakov lower bound on MI to the sequential planning setting, replacing the intractable entropy term with an expectation over an auxiliary distribution $\omega(x|y)$.
  • Conditional Exponential Families: By restricting the auxiliary distribution to the exponential family (conditioned on future observations), we prove that the planning objective becomes convex. This ensures numerical stability and computational efficiency.
  • Relaxed Moment Matching: We show that the optimal parameters for the MI bound arise from a "relaxed" version of the standard exponential family moment-matching property used in variational inference.
The VIP framework involves approximating the posterior (left), the local joint distribution (middle), and the conditional distribution for planning (right).

Experimental Evaluation

We evaluated VIP across three distinct domains:

  1. Nonlinear Sensor Selection: A synthetic 1D target tracking task.
  2. Gene Regulatory Networks: A sparse linear model simulating the discovery of causal gene interactions.
  3. Active Learning for Labeled LDA: Testing the model's ability to learn rare topics using the "Bars" dataset.

In the sensor selection task, VIP achieved state-estimation errors nearly identical to exact numerical methods but at a significantly lower computational cost.

Performance comparison of VIP using different inference methods (EP, ADF) against empirical and random baselines. EP-based planning significantly outperforms ADF-based planning in both state and MI estimate error.

A major insight from our experiments is that the quality of decision-making is heavily dependent on the quality of the posterior approximation. As shown in the performance plots, using Expectation Propagation (EP) led to much better decisions than the simpler Assumed Density Filtering (ADF) because EP provides a more accurate estimate of uncertainty.

Discussion and Key Insights

VIP demonstrates that complex planning problems can be transformed into convex ones through the appropriate choice of auxiliary distributions. This "convexity free lunch" allows for the use of high-speed optimization tools. Furthermore, in the Labeled LDA experiments, VIP outperformed MCMC-based planning. This suggests that the variational bound provides a smoother, more consistent objective than MCMC estimators, which are often biased and prone to local optima.

Limitations and Future Work

The MI bound is only as tight as the auxiliary distribution family allows. If the true relationship between variables is highly complex or multimodal, simple linear Gaussian auxiliary distributions might underestimate information gain. Additionally, if the variational inference method fails to capture the true posterior uncertainty, planning decisions will suffer. Future work will explore applying VIP to time-sensitive domains like autonomous vehicle navigation or real-time signal processing.

Conclusion

Variational Information Planning provides a computationally efficient and robust framework for sequential decision-making. By leveraging variational lower bounds and conditional exponential families, VIP offers a scalable alternative to traditional sampling-based methods, maintaining high accuracy while significantly reducing computational overhead.

Summary updated

Abstract

We consider the setting of sequential decision making where, at each stage, potential actions are evaluated based on expected reduction in posterior uncertainty, given by mutual information (MI). As MI typically lacks a closed form, we propose an approach which maintains variational approximations of, both, the posterior and MI utility. Our planning objective extends an established variational bound on MI to the setting of sequential planning. The result, variational information planning (VIP), is an efficient method for sequential decision making. We further establish convexity of the variational planning objective and, under conditional exponential family approximations, we show that the optimal MI bound arises from a relaxation of the well-known exponential family moment matching property. We demonstrate VIP for sensor selection, experiment design, and active learning, where it meets or exceeds methods requiring more computation, or those specialized to the task.
BibTeX
@inproceedings{pacheco2019variational,
  author = {Jason Pacheco and John W. Fisher III},
  title = {{Variational Information Planning for Sequential Decision Making}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2019},
  url = {https://openreview.net/forum?id=hOsqF49MxtR},
}
Figure