Failing with Grace: Learning Neural Network Controllers that are Boundedly Unsafe

Panagiotis Vlantis, Michael M. ZavlanosarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel method for training neural network controllers for robots that can "fail gracefully." Instead of demanding perfect safety during training, it allows for bounded, controllable safety violations, providing explicit guarantees on the magnitude of these violations. This approach uses reachability analysis to efficiently approximate the robot's future states and penalizes potential unsafe excursions in the training loss.

What Problem Does This Paper Address?

Training neural network controllers for robots that operate in complex, obstacle-filled environments is challenging. Existing methods often rely on extensive, high-quality data near the boundaries of safe operating regions to guarantee safety. This is often impractical to obtain. Furthermore, many approaches focus on ensuring strict invariance of the safe state space, which can be overly conservative or lead to undesirable behaviors. This paper tackles the problem of learning safe robot controllers when perfect safety data is scarce, aiming to achieve a balance between safety and performance by allowing for controlled, bounded deviations from the safe state space.

Methods

The core of the proposed method involves integrating reachability analysis into the neural network training process. Here's a breakdown:

  • Adaptive Cell Partitioning: The robot's state space is adaptively partitioned into cells. Cells that are clearly safe or unsafe are identified, while "mixed" cells (containing both safe and unsafe states) are recursively subdivided. This is done by using over- and under-approximations of the robot's footprint to efficiently determine if a cell is safe. This adaptive approach aims to create a tighter approximation of the safe state space with fewer cells than a uniform grid.
  • Forward Reachable Set Approximation: For each cell in the partitioned safe state space, the method computes an over-approximation of the robot's forward reachable set under the current neural network controller. This is achieved using Interval Bound Propagation (IBP).
  • Safety-Aware Loss Function: A penalty term is added to the standard neural network training loss. This term quantifies the overlap between the approximated forward reachable set of a cell and the set of unsafe states. By minimizing this overlap, the training process is guided to reduce potential safety violations.
  • Iterative Refinement: The process of partitioning the safe state space and refining the neural network controller is performed iteratively. This allows the method to adapt the partition and controller as the controller's safety performance improves.

Datasets

The paper uses a dataset of sampled state-input pairs from safe robot trajectories for initial training. For the numerical experiments, a dataset of 500 feasible trajectories was generated using the RRT* algorithm.

Key Insights and Ideas

  • "Failing with Grace": The central idea is that perfect safety during training is often unattainable. Instead, allowing for bounded and controllable safety violations can lead to more practical and robust controllers.
  • Reachability Analysis for Training: Traditionally used for verification, reachability analysis is repurposed here as a core component of the training loop, directly informing the loss function.
  • Adaptive Partitioning for Efficiency: Instead of a uniform grid, an adaptive cell subdivision strategy is employed. This focuses computational effort on regions near the safety boundary, leading to a more efficient approximation of the safe state space.
  • Trade-off Control: The method explicitly allows for controlling the trade-off between computational complexity and the tightness of safety violation bounds through parameters like the cell subdivision threshold ($\epsilon_w$).

Key Results

  • The proposed method successfully trains neural network controllers that significantly reduce safety violations.
  • In numerical experiments, the total volume of unsafe states decreased by up to 84.6% after re-training.
  • The number of "active" cells (those whose forward reachable sets violate safety) also decreased substantially.
  • The method provides explicit, controllable bounds on safety violations, which are visualized as the overlap between the robot's forward reachable set and unsafe regions.
  • The adaptive partitioning strategy resulted in significantly fewer cells compared to a uniform grid approach for similar safety guarantees.

Limitations and Future Work

  • Computational Cost: While the adaptive partitioning aims for efficiency, reachability analysis can still be computationally intensive, especially for high-dimensional systems or very fine partitions.
  • Assumptions: The method assumes a known, continuously differentiable robot dynamics model and the existence of a function that maps composite intervals of states and inputs to composite intervals of next states.
  • Future Work: The authors suggest extending the method to handle more complex robot dynamics, non-linear activation functions, and potentially dynamic environments. Further exploration of the trade-off between safety and performance is also a promising direction.

Summary updated

Abstract

In this work, we consider the problem of learning a feed-forward neural network controller to safely steer an arbitrarily shaped planar robot in a compact and obstacle-occluded workspace. Unlike existing methods that depend strongly on the density of data points close to the boundary of the safe state space to train neural network controllers with closed-loop safety guarantees, here we propose an alternative approach that lifts such strong assumptions on the data that are hard to satisfy in practice and instead allows for graceful safety violations, i.e., of a bounded magnitude that can be spatially controlled. To do so, we employ reachability analysis techniques to encapsulate safety constraints in the training process. Specifically, to obtain a computationally efficient over-approximation of the forward reachable set of the closed-loop system, we partition the robot's state space into cells and adaptively subdivide the cells that contain states which may escape the safe set under the trained control law. Then, using the overlap between each cell's forward reachable set and the set of infeasible robot configurations as a measure for safety violations, we introduce appropriate terms into the loss function that penalize this overlap in the training process. As a result, our method can learn a safe vector field for the closed-loop system and, at the same time, provide worst-case bounds on safety violation over the whole configuration space, defined by the overlap between the over-approximation of the forward reachable set of the closed-loop system and the set of unsafe states. Moreover, it can control the tradeoff between computational complexity and tightness of these bounds. Our proposed method is supported by both theoretical results and simulation studies.
BibTeX
@misc{vlantis2021failing,
  author = {Panagiotis Vlantis and Michael M. Zavlanos},
  title = {{Failing with Grace: Learning Neural Network Controllers that are Boundedly Unsafe}},
  year = {2021},
  eprint = {2106.11881},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.11881},
}
Figure