Safe Reinforcement Learning in Uncertain Contexts

Dominik Baumann, Thomas B. Schön

Abstract

This paper introduces a framework for safe reinforcement learning (RL) in environments where the "context"—such as a robot's payload or surface friction—is unobserved and must be inferred from noisy sensors like cameras. By deriving new frequentist uncertainty bounds for multi-class classifiers, the authors ensure a robot only proceeds with a task when it is statistically certain of its environment, otherwise triggering a dedicated "identification experiment" to maintain a rigorous safety guarantee.

Introduction

In safe RL, algorithms like SAFEOPT are designed to explore an environment without ever violating safety constraints (e.g., a drone shouldn't crash while learning to fly). However, these algorithms typically assume the "context" of the task is known.

Imagine a robotic arm tasked with balancing a pole. If you add a weight to the pole, the dynamics change. Most safe RL methods assume the robot is told exactly which weight was added. In the real world, the robot might only have a camera. Inferring a discrete property (like weight) from a high-dimensional image is a classification problem fraught with uncertainty. If the classifier is wrong or "over-confident," the robot might choose an unsafe control policy and fail. This paper bridges the gap between sensor-based context inference and provably safe exploration.

Methodology

The authors propose a two-layered approach to identify the current context, integrated into a modified SAFEOPT loop: Sense → Classify → Check Confidence → (Act Safely OR Identify via Experiment).

Experimental setup and decision logic for safe context identification.

1. The Classifier (CME-based)

They use Conditional Mean Embeddings (CMEs) to map sensor measurements (like images) to a probability distribution over possible contexts.

2. Frequentist Uncertainty Bounds

The core theoretical contribution is the derivation of new, input-dependent frequentist bounds for these classifiers. Unlike standard "average-case" accuracy, these bounds tell the robot exactly how much it can trust its prediction for a specific image. This "local" certainty is crucial for safety-critical applications.

3. Maximum Mean Discrepancy (MMD) Fallback

If the image-based classifier is too uncertain (i.e., the lower bound of the probability is below a safety threshold), the robot doesn't guess. Instead, it performs a dedicated "identification experiment" where it moves slightly and uses MMD to compare its current trajectory against recorded trajectories from known contexts. This uses the robot's own physical trajectories as a secondary source of truth.

Experimental Evaluation

The researchers evaluated their approach using three distinct setups:

  • Furuta Pendulum: A real-world and simulated robotic system where the "contexts" were different weights added to the top of a pendulum pole.
  • MNIST: The classic dataset of handwritten digits, used to test the scalability of the new classification bounds.
  • German Traffic Sign Recognition Benchmark (GTSRB): A real-world dataset of traffic signs, used to demonstrate how the algorithm handles safety-critical classification (e.g., a self-driving car needing to be certain of a "Stop" sign).

Key Results and Discussion

  • Zero Failures: In real-world experiments with the Furuta pendulum, the robot successfully learned balancing policies for different weights without the pole ever dropping (the defined "failure" state).
  • Efficiency Gains: Compared to doing a physical identification experiment every time, using the visual classifier significantly reduced the total "training time" required to reach optimal performance.
  • Reliable Uncertainty: On the GTSRB dataset, the algorithm correctly identified when it didn't have enough data to be certain. For example, with only 1,000 training images, the uncertainty intervals were large; with 10,000 images, the intervals shrank, allowing the robot to "trust" its vision.

Conclusion and Future Work

While the framework provides strong safety guarantees, it currently requires an upper bound on the RKHS norm of the true probability function, which is often difficult to estimate and must be chosen heuristically. Future work could extend this to continuous contexts (e.g., any weight between 0 and 100 grams) and address the computational scaling of kernel-based methods as the number of data points grows.

Summary updated

Abstract

When deploying machine learning algorithms in the real world, guaranteeing safety is an essential asset. Existing safe learning approaches typically consider continuous variables, i.e., regression tasks. However, in practice, robotic systems are also subject to discrete, external environmental changes, e.g., having to carry objects of certain weights or operating on frozen, wet, or dry surfaces. Such influences can be modeled as discrete context variables. In the existing literature, such contexts are, if considered, mostly assumed to be known. In this work, we drop this assumption and show how we can perform safe learning when we cannot directly measure the context variables. To achieve this, we derive frequentist guarantees for multiclass classification, allowing us to estimate the current context from measurements. Furthermore, we propose an approach for identifying contexts through experiments. We discuss under which conditions we can retain theoretical guarantees and demonstrate the applicability of our algorithm on a Furuta pendulum with camera measurements of different weights that serve as contexts.
BibTeX
@article{baumann2024safe,
  author = {Dominik Baumann and Thomas B. Schön},
  title = {{Safe Reinforcement Learning in Uncertain Contexts}},
  journal = {IEEE Trans. Robotics},
  year = {2024},
  doi = {10.1109/tro.2024.3354176},
}
Figure