Abstract
Deep Reinforcement Learning (DRL) is a powerful tool for decision-making in Intelligent Cyber-Physical Systems (ICPS), yet it faces significant hurdles due to high-dimensional state spaces and the "black-box" nature of neural networks. This paper introduces a novel abstraction framework that leverages spatio-temporal value semantics to simplify complex systems into verifiable Markov Decision Processes (MDPs). By focusing on behavioral similarity rather than mere physical distance, the proposed method achieves up to a 90% reduction in state space size, accelerates training convergence, and facilitates formal safety verification using probabilistic model checking.
Introduction
In domains like autonomous driving and industrial robotics, the state space is often continuous and infinite, encompassing variables such as velocity, coordinates, and environmental factors. This leads to the "state explosion" problem, where DRL agents struggle with data scarcity and slow convergence. Furthermore, the lack of formal guarantees makes deploying these systems in safety-critical environments risky. Traditional abstraction methods often rely on Euclidean distance, which fails to capture the underlying logic of the task—a "semantic gap" that can lead to inaccurate models.
Methodology
The researchers propose a multi-stage framework to bridge this gap by constructing an abstract MDP that preserves the system's semantics.
Abstraction Process
The framework consists of several key components designed to transform raw traces into a symbolic model:
- Action Abstraction: Continuous action spaces are discretized into finite "Interval Boxes" to ensure mathematical tractability.
- Semantic Interval Abstraction: Utilizing causal discovery algorithms (PC and FCI), the system identifies which state dimensions significantly influence transitions, filtering out noise and spurious correlations.
- Spatio-temporal Value Metric: This core innovation defines state similarity based on immediate rewards, transition probabilities, and long-term value potential.
- Iterative Refinement: A clustering algorithm groups states based on the custom metric, refining granularity until the abstraction error is minimized.
Abstract MDP-Guided DRL
The resulting abstract MDP is not just a verification tool but also a guide for the DRL agent. During training, the agent's actions are influenced by a weighted combination of the neural network's output and the abstract model's recommendations, significantly improving exploration efficiency.
Experimental Evaluation
The approach was tested on three simulated autonomous driving scenarios: Lane Keeping Assist (LKA), Adaptive Cruise Control (ACC), and Intersection Crossroad Assistance (ICA).
Key Results
- State Space Compression: In the ACC scenario, the state space was reduced to approximately 10.12% of its original size without compromising model accuracy.
- Modeling Accuracy: The spatio-temporal metric achieved a Mean Absolute Error (MAE) of 43.43, vastly outperforming the Euclidean-based baseline (MAE of 191.69).
- Training Acceleration: Agents guided by the abstract model reached optimal reward levels faster than standard DRL agents, particularly in the early stages of learning.
- Formal Verification: By converting the system into a symbolic MDP, the researchers successfully used the PRISM model checker to calculate precise safety probabilities, such as the likelihood of collisions or lane departures.
Conclusion
This research demonstrates that incorporating spatio-temporal semantics into the abstraction process can effectively tame the complexity of DRL in ICPS. By narrowing the semantic gap, the framework provides a path toward more efficient training and verifiable safety. Future work will focus on improving the computational efficiency of the abstraction process and exploring formal bisimulation proofs to mathematically guarantee model equivalence.
Summary updated