Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems

Zhehua Zhou, Ozgur S. Oguz, Marion Leibold, Martin BussIEEE TNNLS 2020 · DOI · OpenReview

TL;DR

This paper introduces a data-driven method to learn a low-dimensional representation of a "safe region" for reinforcement learning (RL) in complex, high-dimensional systems. This learned representation helps ensure safety during the RL process by providing reliable estimates of safe states, even when the system dynamics are not perfectly known.

What Problem Does This Paper Address?

Reinforcement learning (RL) has shown great promise in controlling complex systems, but its inherent trial-and-error nature poses a significant safety risk when applied to real-world, high-dimensional dynamical systems. The random exploration during learning can lead to dangerous states that could damage the system or its environment. Existing "Safe RL" (SRL) approaches often rely on simplified models or manual intervention, which are difficult to scale to complex systems. The core challenge is to find a way to guarantee safety during learning without requiring an exact model of the system or excessive human oversight.

Methods

The authors propose a two-part approach:

  1. Data-Driven Initialization of the Safe Region: Instead of relying on hand-crafted physics-based models for "model order reduction," they use a data-driven method. They leverage an approximated system model (which is easier to obtain) to generate training data. This data is then processed using a modified t-distributed stochastic neighbor embedding (t-SNE) algorithm. The key innovation here is a custom distance metric that considers both trajectory similarity and safety labels (whether the system recovered safely or not). This allows them to learn a low-dimensional "safety feature" and a corresponding "Safety Assessment Function" (SAF). Belief function theory is used to represent the uncertainty in these safety estimates.
  2. Online Adaptation of the Safe Region: To bridge the gap between the approximated model (simulation) and the real system (reality), they introduce an online adaptation mechanism. As the RL agent interacts with the real system, feedback data is collected. This feedback is used to update the initial safety estimates. Specifically, they use a Gaussian Process Regression (GPR) model to predict the consistency of safety between simulation and reality for observed states. This prediction, along with its confidence, is used to refine the belief masses in the belief function framework, making the safety estimates more accurate over time.

Datasets

  • Training Dataset (Dtrain): Generated from a nominal (approximated) quadcopter system model. This dataset contains initial states, safety labels (whether the corrective controller successfully recovered the system), and the corresponding system trajectories.
  • Feedback Dataset (Dfeedback): Collected from the real quadcopter system during the learning process. This dataset also contains initial states, safety labels, and trajectories, but from the actual system.

Key Insights and Ideas

  • Data-Driven Safety, Not Physics-Driven: The core idea is to learn safety representations from data rather than relying on potentially complex and inaccurate physics-based simplifications. This makes the approach more generalizable.
  • Safety is Probabilistic: The approach doesn't aim for absolute safety guarantees but rather probabilistic estimates of safety, which is more realistic for complex, uncertain systems.
  • Bridging Simulation and Reality: The online adaptation step is crucial. It acknowledges the "simulation-to-reality gap" and actively uses real-world feedback to correct and refine the safety estimates learned from simulation.
  • Belief Function Theory for Uncertainty: Using belief functions provides a principled way to represent and fuse uncertain safety information from different sources (training data and feedback data).

Key Results

The paper demonstrates the effectiveness of their approach on a quadcopter example.

  • Improved Safe Region Representation: The learned low-dimensional representation of the safe region is shown to be more reliable and representative compared to previous methods, particularly those relying on physically inspired model order reduction.
  • Clearer Separation of Safe/Unsafe States: Visualizations show that the proposed method achieves a better separation between safe and unsafe states in the learned low-dimensional space, even for complex scenarios.
  • Effective Online Adaptation: The online adaptation method successfully refines the safety estimates as more feedback data is collected, leading to more accurate DSAFs (Discretized Safety Assessment Functions).

Limitations and Future Work

  • Sensitivity to Nominal Model Quality: The performance of the approach is still affected by the quality of the initial approximated system model. A large discrepancy between the nominal and real systems can lead to less representative safety features.
  • Future Work: The authors suggest combining data-driven methods with model-based techniques for even greater robustness to the simulation-to-reality gap. They also aim to investigate generalizing the learned safety features across different but similar dynamical systems.

Summary updated

Abstract

For safely applying reinforcement learning algorithms on high-dimensional nonlinear dynamical systems, a simplified system model is used to formulate a safe reinforcement learning framework. Based on the simplified system model, a low-dimensional representation of the safe region is identified and is used to provide safety estimates for learning algorithms. However, finding a satisfying simplified system model for complex dynamical systems usually requires a considerable amount of effort. To overcome this limitation, we propose in this work a general data-driven approach that is able to efficiently learn a low-dimensional representation of the safe region. Through an online adaptation method, the low-dimensional representation is updated by using the feedback data such that more accurate safety estimates are obtained. The performance of the proposed approach for identifying the low-dimensional representation of the safe region is demonstrated with a quadcopter example. The results show that, compared to previous work, a more reliable and representative low-dimensional representation of the safe region is derived, which then extends the applicability of the safe reinforcement learning framework.
BibTeX
@article{zhou2020learning,
  author = {Zhehua Zhou and Ozgur S. Oguz and Marion Leibold and Martin Buss},
  title = {{Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2020},
  doi = {10.1109/tnnls.2021.3106818},
}
Figure