PACSBO: Probably approximately correct safe Bayesian optimization

Abdullah Tokmak, Thomas B. Schön, Dominik BaumannarXiv 2024 · DOI · arXiv

Abstract

Safe Bayesian Optimization (BO) typically requires a known upper bound on a function's smoothness (the RKHS norm) to guarantee safety—a value that is nearly impossible to determine in the real world. PACSBO eliminates this "guessing game" by estimating this bound directly from data with probabilistic guarantees and using a localized approach to reduce over-cautious exploration.

Introduction

In robotics and control, finding optimal parameters for complex systems like drones or pendulums requires balancing performance with safety. Safe Bayesian Optimization (BO) addresses this by building a Gaussian Process (GP) model to navigate the parameter space while respecting safety boundaries. However, a significant bottleneck exists: algorithms like SafeOpt require a known upper bound on the Reproducing Kernel Hilbert Space (RKHS) norm.

Visualizing the RKHS norm as a measure of function 'wiggliness' or complexity.

The RKHS norm quantifies function complexity. If the bound is set too high, the algorithm becomes overly conservative; if set too low, it risks catastrophic safety violations. In practice, engineers often resort to guessing, which compromises theoretical safety guarantees.

Methods

PACSBO introduces a principled framework to estimate these bounds dynamically through three main innovations.

RKHS Norm Estimation

The core insight is that as sampling density increases, the RKHS norm of the GP mean function converges to the true norm of the underlying function. PACSBO utilizes a Multilayer Perceptron (MLP) to predict the true norm based on the current mean function and a "sampling density" metric ($r_{A,A}$), which quantifies how well the space has been explored.

The RKHS norm of the GP mean function approaches the true norm as sampling density ($r_{A,A}$) increases.

PAC Guarantees and Locality

To ensure safety, the MLP's estimate is augmented with a safety factor derived via Hoeffding’s inequality. By generating random RKHS functions that interpolate current data, the algorithm provides Probably Approximately Correct (PAC) guarantees that the estimated bound is a true over-estimation with probability $1-\delta$.

Furthermore, PACSBO treats smoothness as a local property. By partitioning the parameter space into sub-domains (the sample convex hull, a buffer zone, and the global space), the agent can explore more aggressively in regions confirmed to be smooth, rather than being constrained by a global worst-case bound.

Results

The algorithm was validated using synthetic Matérn-32 functions and real-world hardware.

  • Efficiency: PACSBO explored safe regions significantly faster than SafeOpt when the latter was given conservative manual bounds.
  • Safety: Unlike SafeOpt with optimistic bounds, PACSBO successfully avoided constraint violations by updating its complexity estimates as it encountered more "wiggly" data.
  • Hardware Validation: On a Furuta Pendulum, PACSBO tuned a state-feedback controller in just 15 samples without any safety failures or prior knowledge of the system's RKHS norm.

Conclusion

PACSBO bridges the gap between the theoretical requirements of safe BO and practical implementation. By estimating function complexity from data and applying local smoothness assumptions, it provides a robust framework for safe autonomous learning. Future work includes more adaptive space partitioning and improving computational scalability for high-dimensional tasks.

Summary updated

Abstract

Safe Bayesian optimization (BO) algorithms promise to find optimal control policies without knowing the system dynamics while at the same time guaranteeing safety with high probability. In exchange for those guarantees, popular algorithms require a smoothness assumption: a known upper bound on a norm in a reproducing kernel Hilbert space (RKHS). The RKHS is a potentially infinite-dimensional space, and it is unclear how to, in practice, obtain an upper bound of an unknown function in its corresponding RKHS. In response, we propose an algorithm that estimates an upper bound on the RKHS norm of an unknown function from data and investigate its theoretical properties. Moreover, akin to Lipschitz-based methods, we treat the RKHS norm as a local rather than a global object, and thus reduce conservatism. Integrating the RKHS norm estimation and the local interpretation of the RKHS norm into a safe BO algorithm yields PACSBO, an algorithm for probably approximately correct safe Bayesian optimization, for which we provide numerical and hardware experiments that demonstrate its applicability and benefits over popular safe BO algorithms.
BibTeX
@misc{tokmak2024pacsbo,
  author = {Abdullah Tokmak and Thomas B. Schön and Dominik Baumann},
  title = {{PACSBO: Probably approximately correct safe Bayesian optimization}},
  year = {2024},
  eprint = {2409.01163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.01163},
}
Figure