Unbounded Bayesian Optimization via Regularization

Bobak Shahriari, Alexandre Bouchard-Côté, Nando de FreitasAISTATS 2016 · OpenReview

Abstract

Bayesian optimization has recently emerged as a powerful and flexible tool in machine learning for hyperparameter tuning and more generally for the efficient global optimization of expensive black box functions. The established practice requires a user-defined bounded domain, which is assumed to contain the global optimizer. However, when little is known about the probed objective function, it can be difficult to prescribe such a domain. In this work, we modify the standard Bayesian optimization framework in a principled way to allow for unconstrained exploration of the search space. We introduce a new alternative method and compare it to a volume doubling baseline on two common synthetic benchmarking test functions. Finally, we apply our proposed methods on the task of tuning the stochastic gradient descent optimizer for both a multi-layered perceptron and a convolutional neural network on the MNIST dataset.

Introduction

Standard Bayesian Optimization (BO) is typically confined within user-defined bounding boxes. These boundaries are often set arbitrarily, leading to a "Catch-22": a box that is too small may exclude the global optimum, while one that is too large forces the algorithm to waste resources on irrelevant regions. Technically, common acquisition functions like Expected Improvement (EI) can have maxima at infinity, necessitating these hard constraints to prevent the algorithm from suggesting points infinitely far away. This work proposes replacing these "hard" walls with "soft" regularization penalties, enabling safe and effective unconstrained exploration.

Methodology: Regularization via Prior Means

The core contribution of this paper is a principled regularization approach using non-stationary Gaussian Process (GP) priors. Instead of a constant prior mean, the authors introduce a non-stationary prior mean that acts as a regularizer, ensuring the acquisition function decays to zero as it approaches infinity.

Two primary strategies are compared:

  1. Quadratic (EI-Q): Employs a parabolic prior mean centered on the initial search region.
  2. Hinge-Quadratic (EI-H): The preferred method, which maintains a standard flat prior mean within a radius $R$ and applies a quadratic penalty only outside this "trusted" region.
Comparison of Hinge-Quadratic and Quadratic regularization views. The Hinge-Quadratic (EI-H) preserves the acquisition function's shape within the trusted region while providing a safety rail outside.

This regularization allows practitioners to use unconstrained optimizers for finding the next sample point, effectively permitting the algorithm to wander outside the initial guess if the data indicates a better result lies elsewhere.

Experimental Evaluation

The methods were evaluated on synthetic benchmarks and real-world machine learning tasks:

  • Synthetic Functions: Hartmann 3 and Hartmann 6 functions. A "Hartmann*" variant was created where the global optimum was intentionally placed outside the initial box to test exploration.
  • Real-World ML: Tuning Stochastic Gradient Descent (SGD) hyperparameters (learning rate, momentum, $L_1/L_2$ regularization) for a Multi-Layered Perceptron (MLP) and a Convolutional Neural Network (CNN) on the MNIST dataset.

Insights and Results

A key observation is the "Boundary Bias" in standard BO, where samples often cluster at the corners or edges of the bounding box. Regularization smooths this behavior, leading to more homogeneous search patterns.

Visualization of search points expanding beyond the initial domain over iterations. The algorithm progressively explores outside the initial region as the number of samples $n$ increases.

Key Findings:

  • Finding Hidden Optima: In Hartmann* experiments, EI-H successfully located the global optimum outside the box, whereas standard BO remained trapped at the boundary.
  • Performance Gains: On the MNIST MLP task, EI-H achieved over a 20% improvement in test accuracy compared to standard BO by identifying an optimal learning rate outside the initial human-provided range.
  • Robustness: While a simple "Volume Doubling" (EI-V) heuristic performed well, the regularized EI-H was more consistent and avoided the artifacts associated with box-based methods.

Conclusion

By shifting from hard boundaries to soft regularization, Bayesian Optimization can be made more robust to poorly specified search domains. While challenges like the curse of dimensionality remain, this approach provides a user-friendly "safety rail" that respects initial guesses without being imprisoned by them. Future work may explore more complex non-stationary kernels to further enhance global search capabilities.

Summary updated

Abstract

Bayesian optimization has recently emerged as a powerful and flexible tool in machine learning for hyperparameter tuning and more generally for the efficient global optimization of expensive black box functions. The established practice requires a user-defined bounded domain, which is assumed to contain the global optimizer. However, when little is known about the probed objective function, it can be difficult to prescribe such a domain. In this work, we modify the standard Bayesian optimization framework in a principled way to allow for unconstrained exploration of the search space. We introduce a new alternative method and compare it to a volume doubling baseline on two common synthetic benchmarking test functions. Finally, we apply our proposed methods on the task of tuning the stochastic gradient descent optimizer for both a multi-layered perceptron and a convolutional neural network on the MNIST dataset.
BibTeX
@inproceedings{shahriari2016unbounded,
  author = {Bobak Shahriari and Alexandre Bouchard-Côté and Nando de Freitas},
  title = {{Unbounded Bayesian Optimization via Regularization}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2016},
  url = {https://openreview.net/forum?id=HwVc74Nyhh},
}
Figure