Safe Exploration for Interactive Machine Learning

Matteo Turchetta, Felix Berkenkamp, Andreas Krause

TL;DR

The authors introduce GOOSE (Goal-Oriented Safe Exploration), a framework that transforms existing "unsafe" machine learning algorithms into safe ones. By focusing exploration only on the areas necessary to reach a specific goal—rather than mapping out every possible safe corner of an environment—GOOSE achieves significantly higher data efficiency while maintaining rigorous safety guarantees.

What Problem Does This Paper Address?

Interactive Machine Learning (IML) involves an agent (like a robot or a tuning algorithm) making decisions, observing outcomes, and updating its strategy. In the real world, some decisions are catastrophic—a drone crashing or a chemical reactor overheating.

Standard IML algorithms often ignore these "safety constraints" during the learning phase. Existing "safe" algorithms, like SAFEOPT, solve this by meticulously mapping out the entire "safe set" before trying to optimize. However, this is incredibly slow. Imagine trying to drive to a specific grocery store, but refusing to leave your driveway until you have personally mapped every safe street in the entire city. This paper addresses this sample inefficiency by proposing a goal-directed approach to safety.

Methods

The researchers developed GOOSE, an "add-on" module that sits between an existing IML algorithm (the "oracle") and the environment.

GOOSE Flowchart
  1. The Oracle: GOOSE can work with any standard, potentially unsafe algorithm (like GP-UCB for Bayesian Optimization). The oracle suggests a target decision $x^*$.
  2. Pessimistic vs. Optimistic Sets: GOOSE maintains two views of the world based on Gaussian Process (GP) models:
  3. Pessimistic Safe Set: Areas where the algorithm is statistically certain it is safe to operate.
  4. Optimistic Safe Set: Areas that could be safe based on current knowledge.
  5. Goal-Oriented Expansion: If the oracle’s suggestion is already in the pessimistic safe set, it is executed immediately. If it is outside, GOOSE doesn't just give up; it identifies a path of "learning targets" that could eventually prove the suggestion is safe. It then evaluates the most informative points along that specific path.
  6. Regularity Assumptions: The method relies on the assumption that the safety constraint function is "smooth" (specifically, it has a bounded norm in a Reproducing Kernel Hilbert Space), allowing the agent to use nearby observations to infer the safety of unvisited points.

Key Insights and Ideas

  • Don't Map the Whole World: The core insight is that safety exploration should be a means to an end, not the end itself. By tying exploration to the oracle's suggestions, the agent ignores safe areas that are irrelevant to the current task.
  • Modularity: Because GOOSE acts as an add-on, it is "algorithm-agnostic." You can take the latest optimization algorithm and "wrap" it in GOOSE to make it safe without redesigning the underlying logic.
  • Provable Safety: Despite being more aggressive and efficient than previous methods, the authors provide a mathematical proof that GOOSE will not violate safety constraints (with high probability) and will eventually find the best reachable solution.

Key Results

To evaluate GOOSE, the authors tested it across synthetic GP samples, synthetic grid worlds, and Mars exploration data (using real topography from the Mars Reconnaissance Orbiter).

Comparison of STAGEOPT and GOOSE
  • Faster Convergence: In Bayesian Optimization tasks, GOOSE reached the global optimum much faster than SAFEOPT or STAGEOPT because it didn't waste evaluations on the boundaries of the safe set that were far from the peak.
  • Efficiency in MDPs: In the Mars rover simulations, the rover needed to find a safe path while avoiding slopes steeper than 30°. GOOSE found the first safe path to the goal using 2.5x fewer samples than the standard Safe MDP (SMDP) approach.
  • Computational Performance: While more complex than a simple "unsafe" algorithm, GOOSE's overhead was negligible compared to the cost of real-world evaluations, and it was significantly faster to compute than competing safe reinforcement learning algorithms like SEO.

Limitations and Future Work

  • The Lipschitz Constant: Like many GP-based safe methods, GOOSE performs best when the "smoothness" (Lipschitz constant) of the safety constraint is known. In practice, estimating this constant accurately remains a challenge.
  • Initial Safe Seed: The algorithm requires a small "seed" of known safe points to start. It cannot safely explore a completely unknown environment from a potentially unsafe starting position.
  • Deterministic Environments: While the paper touches on Markov Decision Processes (MDPs), the current analysis focuses on deterministic transitions. Extending the same level of goal-oriented efficiency to highly stochastic environments is a logical next step.

Summary updated

Abstract

In interactive machine learning (IML), we iteratively make decisions and obtain noisy observations of an unknown function. While IML methods, e.g., Bayesian optimization and active learning, have been successful in applications, on real-world systems they must provably avoid unsafe decisions. To this end, safe IML algorithms must carefully learn about a priori unknown constraints without making unsafe decisions. Existing algorithms for this problem learn about the safety of all decisions to ensure convergence. This is sample-inefficient, as it explores decisions that are not relevant for the original IML objective. In this paper, we introduce a novel framework that renders any existing unsafe IML algorithm safe. Our method works as an add-on that takes suggested decisions as input and exploits regularity assumptions in terms of a Gaussian process prior in order to efficiently learn about their safety. As a result, we only explore the safe set when necessary for the IML problem. We apply our framework to safe Bayesian optimization and to safe exploration in deterministic Markov Decision Processes (MDP), which have been analyzed separately before. Our method outperforms other algorithms empirically.
BibTeX
@inproceedings{turchetta2019safe,
  author = {Matteo Turchetta and Felix Berkenkamp and Andreas Krause},
  title = {{Safe Exploration for Interactive Machine Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=OYgwQejiHCB},
}
Figure