Gaussian Process Bandit Optimization of the Thermodynamic Variational Objective

Vu Nguyen, Vaden Masrani, Rob Brekelmans, Michael A. Osborne, Frank Wood

Abstract

This paper introduces a Gaussian process (GP) bandit optimization method to automatically and dynamically tune the "integration schedule" for the Thermodynamic Variational Objective (TVO). By framing schedule selection as a regret-minimization problem, the authors achieve superior model learning and inference in Variational Autoencoders (VAEs) while eliminating the need for expensive manual grid searches. The method demonstrates significant efficiency gains, achieving competitive results on benchmarks like Omniglot in a fraction of the time required for traditional tuning.

Introduction

In deep generative modeling, Variational Autoencoders (VAEs) typically optimize the Evidence Lower Bound (ELBO). While widely used, the ELBO often provides a loose bound on the true log-likelihood. The Thermodynamic Variational Objective (TVO) offers a more powerful alternative by framing log-likelihood estimation as a one-dimensional Riemann integral over a geometric mixture curve.

The TVO objective frames log-likelihood estimation as a Riemann sum approximation to a 1-d integral.

As illustrated in Figure 1, the TVO (represented by the blue area) provides a tighter bound than the ELBO (the area within the dotted lines). However, the effectiveness of the TVO depends critically on the "integration schedule"—the specific set of discretization points $\beta$ used for the Riemann sum. Historically, finding an optimal schedule required static heuristics or exhaustive grid searches. Furthermore, the optimal schedule is non-stationary; it evolves as the model trains, making static schedules inherently suboptimal.

Methodology: GP Bandit Optimization

The authors propose automating schedule selection using Gaussian Process Bandit Optimization. They treat the choice of the schedule $\beta$ as a multi-armed bandit problem within a continuous space. Key innovations in this approach include:

  • Time-Varying Reward Function: The "reward" for a chosen schedule is defined as the improvement in the TVO objective over a window of training epochs. Since the model's integrand changes during training, a time-varying Gaussian Process is employed to model this shifting reward landscape.
  • Permutation-Invariant Kernel: Integration points in a Riemann sum are ordered ($0 < \beta_1 < ... < \beta_{d-1} < 1$). The authors utilize a bespoke GP kernel with a sorting projection to ensure the model recognizes that different permutations of the same points represent the same schedule.
  • Dynamic Adaptation: The algorithm periodically selects a new schedule $\beta$ by maximizing an acquisition function (GP-UCB), effectively balancing the exploration of new schedules with the exploitation of known high-performing regions.

Experimental Results

The method was evaluated across several standard benchmarks, including continuous VAEs (MNIST, Fashion MNIST), discrete latent variable models (SBNs on binarized MNIST and Omniglot), and structured models (Astronomers PCFG).

Key Insights

  • Regret Minimization as Evidence Maximization: The authors provide a theoretical proof that minimizing the cumulative regret of the bandit algorithm is equivalent to maximizing the final log evidence of the generative model.
  • Evolution of Scheduling: The bandit's behavior typically follows three distinct phases: initial random exploration, a focus on small $\beta$ values where the integrand is steepest early in training, and a shift toward $\beta=1$ as the model matures and the integrand smooths out.
Bandit-chosen integration points over time, showing the transition from exploration to exploitation.

Performance Gains

The GP-bandit approach consistently outperformed standard baselines, such as log-uniform or "moments" schedules, in both model learning (test log-likelihood) and inference quality (test KL divergence). Most notably, the method offers a significant speedup: on the Omniglot dataset, it achieved competitive results in just 11 hours, compared to the 177 hours required for a comprehensive grid search—a roughly 16x improvement in efficiency.

Conclusion and Future Work

By automating the integration schedule of the TVO, this work removes a significant barrier to using more powerful variational objectives. While the positions of the integration points are now dynamic, the number of partitions ($d$) remains a fixed hyperparameter. Future research may explore adaptively selecting the dimensionality of the schedule itself. Additionally, while computationally efficient compared to grid search, further optimization could reduce the overhead of fitting the GP in extremely high-dimensional latent spaces.

Summary updated

Abstract

Achieving the full promise of the Thermodynamic Variational Objective (TVO), a recently proposed variational lower bound on the log evidence involving a one-dimensional Riemann integral approximation, requires choosing a "schedule" of sorted discretization points. This paper introduces a bespoke Gaussian process bandit optimization method for automatically choosing these points. Our approach not only automates their one-time selection, but also dynamically adapts their positions over the course of optimization, leading to improved model learning and inference. We provide theoretical guarantees that our bandit optimization converges to the regret-minimizing choice of integration points. Empirical validation of our algorithm is provided in terms of improved learning and inference in Variational Autoencoders and Sigmoid Belief Networks.
BibTeX
@inproceedings{nguyen2020gaussian,
  author = {Vu Nguyen and Vaden Masrani and Rob Brekelmans and Michael A. Osborne and Frank Wood},
  title = {{Gaussian Process Bandit Optimization of the Thermodynamic Variational Objective}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=kShU4LS5Bza},
}
Figure