Data Generation Method for Learning a Low-dimensional Safe Region in Safe Reinforcement Learning

Zhehua Zhou, Ozgur S. Oguz, Yi Ren, Marion Leibold, Martin BussarXiv 2021 · OpenReview

TL;DR

This paper tackles the challenge of making Reinforcement Learning (RL) safe for complex, high-dimensional systems by learning a low-dimensional "safe region." They propose a novel data generation method that combines uniform and multivariate normal distributions to create training data, balancing learning performance with safety guarantees.

What Problem Does This Paper Address?

Reinforcement Learning (RL) has shown great promise in controlling complex dynamical systems. However, a major hurdle for applying RL to real-world systems is safety. During the learning process, an RL agent explores different actions, which can lead to unsafe states that damage the system or environment. Safe Reinforcement Learning (SRL) aims to mitigate this by ensuring safety during learning. A common approach in SRL is to define a "safe region" in the system's state space. If the agent stays within this region, it's guaranteed to be controllable back to a safe state. However, for highly nonlinear and high-dimensional systems, defining this safe region directly is computationally infeasible. This paper addresses this by proposing a data-driven method to learn a low-dimensional representation of this safe region, making SRL more practical for complex systems. The core problem they investigate is how the quality and characteristics of the training data used to learn this safe region impact the overall SRL performance.

Methods

The paper builds upon a previously proposed SRL framework that uses a low-dimensional representation of the safe region. This framework involves:

  1. State Mapping: Mapping high-dimensional system states ($x \in \mathbb{R}^n$) to a lower-dimensional "safety feature" ($y \in \mathbb{R}^{n_y}$, where $n_y < n$) using a function $\Psi(x)$.
  2. Safety Assessment: Estimating the probability of a state being safe using a learned function $\Gamma(y)$.
  3. Simplified Safe Region ($S_y$): Defining a safe region in the low-dimensional space based on a probability threshold ($p_t$).
  4. Supervisory Control: A supervisor switches between the learning-based controller ($\pi(x)$) and a corrective controller ($K(x)$) to keep the system within $S_y$.

The novel contribution is a data generation method for creating the training dataset ($D_{tr}$) used to learn $\Psi(x)$ and $\Gamma(y)$. Instead of relying solely on a uniform distribution (UD) or a multivariate normal distribution (MND) of system states, they propose a combined approach:

$D_{tr} = D_{UD} + D_{MND}$

  • $D_{UD}$ (Uniform Distribution): Samples states uniformly across the entire state space. This tends to make the learned safe region more conservative, prioritizing safety over exploration.
  • $D_{MND}$ (Multivariate Normal Distribution): Samples states from a distribution fitted to trajectories generated by a nominal system model. This distribution is typically concentrated around the origin (a known safe state), leading to less conservative safety estimates.

The parameter $\alpha$ controls the balance between these two distributions, allowing a trade-off between learning performance and safety.

Datasets

  • Training Data: The paper generates its own training datasets. They use two primary sampling strategies:
  • Uniform Distribution (UD): States sampled uniformly across the state space.
  • Multivariate Normal Distribution (MND): States sampled from a distribution fitted to trajectories of a nominal system model.
  • Evaluation Data: The performance is evaluated using a three-link inverted pendulum system. The "real" system is simulated with slightly perturbed parameters (masses) compared to the nominal system used for generating $D_{MND}$.

Key Insights and Ideas

  • Data Distribution is Crucial for Safety: The distribution of training data significantly impacts the learned safety estimates and, consequently, the overall SRL performance. A purely uniform distribution leads to conservative safety regions, potentially hindering learning, while a purely MND-based distribution can be too optimistic and lead to unsafe exploration.
  • Balancing Act: The proposed combined data generation method offers a principled way to balance the trade-off between learning a good policy (high reward) and ensuring system safety (low number of failures/conservatism).
  • Conservatism as a Safety Feature: A more conservative safe region (achieved with higher $\alpha$, favoring UD) leads to more frequent activation of the corrective controller but also a higher probability of the system remaining safe. This conservatism is acceptable as long as a satisfactory policy can still be learned.
  • Nominal Model's Role: The nominal system model is used to generate a distribution ($D_{MND}$) that reflects typical system behavior, providing a less conservative baseline for safety estimation. The discrepancy between this nominal model and the real system is a key factor influencing the effectiveness of the learned safety estimates.

Key Results

The paper demonstrates the effectiveness of their combined data generation method on a three-link inverted pendulum task.

  • Learning Performance: Using the combined approach (especially with $\alpha=0.5$) generally resulted in higher rewards compared to using only the uniform distribution ($\alpha=1$), which led to overly conservative behavior and lower rewards. The free learning case (no supervisor) resulted in failures.
  • Safety: The combined approach and the UD-only approach significantly reduced failures and the number of corrective controller activations compared to free learning. The UD-only approach ($\alpha=1$) was the most conservative, ensuring the highest safety, but at the cost of learning performance.
  • Discrepancy Impact: The method showed effectiveness across different levels of discrepancy between the nominal and real systems ($\Delta = 1.1, 1.5, 4$). The UD-based supervisor was generally more robust to mismatches due to its higher inherent conservatism.

Limitations and Future Work

  • Data-Driven Reliance: The reliability of the learned safety estimates is inherently data-dependent. The performance is tied to the quality and representativeness of the training data.
  • Computational Infeasibility: While the method aims to overcome the computational infeasibility of defining safe regions for high-dimensional systems, the initial generation of data from a nominal system and the subsequent t-SNE embedding still involve computational steps.
  • Future Work: The authors suggest finding a metric to quantify the discrepancy between the nominal and real systems, which could then be used to dynamically guide the data generation process. They also aim to explore how different data generation strategies affect the reliability of safety estimates in general data-driven safety prediction methods.

Summary updated

Abstract

Safe reinforcement learning aims to learn a control policy while ensuring that neither the system nor the environment gets damaged during the learning process. For implementing safe reinforcement learning on highly nonlinear and high-dimensional dynamical systems, one possible approach is to find a low-dimensional safe region via data-driven feature extraction methods, which provides safety estimates to the learning algorithm. As the reliability of the learned safety estimates is data-dependent, we investigate in this work how different training data will affect the safe reinforcement learning approach. By balancing between the learning performance and the risk of being unsafe, a data generation method that combines two sampling methods is proposed to generate representative training data. The performance of the method is demonstrated with a three-link inverted pendulum example.
BibTeX
@misc{zhou2021data,
  author = {Zhehua Zhou and Ozgur S. Oguz and Yi Ren and Marion Leibold and Martin Buss},
  title = {{Data Generation Method for Learning a Low-dimensional Safe Region in Safe Reinforcement Learning}},
  year = {2021},
  eprint = {2109.05077},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.05077},
}
Figure