STL-Based Synthesis of Feedback Controllers Using Reinforcement Learning

Nikhil Kumar Singh, Indranil SahaAAAI 2023 · DOI

TL;DR

This paper tackles the challenge of synthesizing safe and robust feedback controllers for complex cyber-physical systems using Deep Reinforcement Learning (DRL). They introduce a novel Signal Temporal Logic (STL) semantics that is smooth and addresses the "shadowing problem," enabling more effective reward generation for DRL agents, leading to superior controller performance on challenging benchmarks.

What Problem Does This Paper Address?

Designing feedback controllers for safety-critical cyber-physical systems (CPSs) is crucial, but traditional methods rely on accurate mathematical models, which are often difficult to obtain for complex systems. Reinforcement Learning (RL), particularly Deep Reinforcement Learning (DRL), offers a promising alternative by learning controllers through interaction with the system. However, a key challenge in RL is designing effective reward functions that accurately capture desired system behaviors, especially complex safety and liveness properties often expressed using temporal logic. Existing approaches to using temporal logic for reward generation have limitations: classical semantics can lead to suboptimal controllers due to the "shadowing problem," while other aggregation-based semantics are not smooth, hindering gradient-based RL. This paper aims to bridge this gap by proposing a new STL semantics that is both smooth and addresses the shadowing problem, making it suitable for DRL-based controller synthesis.

Methods

The core contribution of this paper is the introduction of a new aggregation-based semantics for Signal Temporal Logic (STL) called Smoothened-Summation-Subtraction (SSS). This semantics is designed to be used as a reward generation mechanism within a DRL framework.

Key aspects of their methodology include:

  • Novel STL Semantics (SSS): The SSS semantics is proposed to overcome the limitations of existing STL semantics. It is designed to be:
  • Smooth: This is crucial for gradient-based RL algorithms, as it leads to smoother policies and faster learning. They achieve smoothness by approximating non-smooth functions (like the absolute value and max/min operators) with smooth alternatives (e.g., using the error function and logarithmic summations).
  • Shadow-lifting: It addresses the "shadowing problem" where the robustness of individual sub-specifications doesn't always contribute to the overall robustness in a desirable way.
  • Min-max Bounded: The output of the semantics is bounded by the minimum and maximum of the individual sub-specification robustness values.
  • Not Sound (by design): While not strictly "sound" in the traditional sense (meaning a negative robustness doesn't always guarantee violation), the authors argue this is acceptable for reward generation in RL, as it can still guide the agent towards desirable behaviors. They also introduce "aggregate soundness" to characterize its behavior.
  • DRL Framework: They integrate their SSS semantics into a standard Actor-Critic DRL algorithm (specifically, Soft Actor-Critic - SAC). The STL robustness computed online serves as the reward signal for the RL agent.
  • Online Monitoring: The STL specifications are monitored online, meaning the robustness is computed incrementally as the system interacts, which is essential for real-time control.
  • Policy Gradient: They utilize policy gradient methods to optimize the controller's parameters, with the Q-function (critic) leveraging the online STL robustness.

Datasets

The paper evaluates their approach on several standard continuous control benchmarks from the OpenAI Gym environment. These include:

  • HalfCheetah (HC)
  • Hopper (Hop)
  • Ant
  • Walker (Wkr)
  • Swimmer (Swm)
  • Humanoid (Hum)

No new benchmarks were introduced; instead, they used existing environments and defined STL specifications tailored to capture the safety and liveness requirements of each environment.

Key Insights and Ideas

  • Smoothness is Key for DRL Rewards: The paper strongly advocates for smooth reward functions in DRL. They demonstrate that non-smooth semantics, even if theoretically sound, can hinder learning efficiency and lead to suboptimal controllers.
  • Soundness is Not Always Necessary for Reward Generation: The authors challenge the strict requirement of soundness for STL semantics when used as rewards. Their SSS semantics, which is not sound but is smooth and addresses shadowing, proves more effective in practice. This suggests a trade-off between theoretical guarantees and practical performance in RL.
  • Addressing the Shadowing Problem is Crucial: The "shadowing problem" in classical STL semantics can lead to situations where improving one part of a specification doesn't improve the overall satisfaction. Their SSS semantics effectively mitigates this, leading to better overall controller behavior.
  • STL as a Powerful Specification Language for RL: The work highlights the potential of using formal temporal logic specifications (like STL) to guide complex RL tasks, moving beyond hand-crafted reward functions.

Key Results

The paper presents experimental results comparing their SSS semantics with other state-of-the-art STL semantics (Classical, AGM, LSE, Softmax) across various benchmarks.

  • Superior Performance: The SSS semantics consistently outperformed other semantics across most metrics.
  • Synthesizing Useful Controllers: SSS was the only semantics capable of synthesizing a useful controller for all tested benchmarks, particularly for the Swimmer environment where other methods failed.
  • Maximized Distance Covered (DC): Controllers synthesized using SSS achieved the maximum distance covered among all tested semantics.
  • High Safety Satisfaction (SAT): SSS controllers generally achieved high safety satisfaction (often 100%), indicating robust adherence to safety constraints.
  • Trade-off with Control Cost (CC): While SSS controllers were not always the cheapest in terms of control effort, this was often a necessary trade-off for achieving better performance (higher DC, MoS, SAT). Low CC from other semantics sometimes indicated the robot was not moving effectively.
  • High Margin of Satisfaction (MoS): SSS generally achieved the highest MoS, indicating a good balance between satisfying safety and liveness properties.

Limitations and Future Work

  • Not Sound: The authors acknowledge that their SSS semantics is not sound. While they argue this is acceptable for reward generation, it means that a negative robustness value from SSS doesn't strictly guarantee a violation of the specification.
  • Tunable Parameters: The SSS semantics involves tunable parameters (μ and η) that influence its behavior. While they found suitable values through initial experiments, their optimal selection might be environment-dependent.
  • Future Work: The authors plan to explore the application of their semantics in other complex domains, such as self-driving cars. They also suggest further investigation into the theoretical properties and practical implications of their aggregate soundness concept.

Summary updated

BibTeX
@inproceedings{singh2023stl,
  author = {Nikhil Kumar Singh and Indranil Saha},
  title = {{STL-Based Synthesis of Feedback Controllers Using Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26764},
}
Figure