Enhancing safety and explainability of reinforcement learning agents for environmental monitoring tasks

Introduction

Monitoring water quality and biodiversity in aquatic ecosystems is a critical yet resource-intensive task. While autonomous aquatic drones offer a scalable solution, traditional Deep Reinforcement Learning (DRL) agents often operate as "black boxes," prone to safety violations and sample inefficiency. Researchers from the University of Verona have introduced a novel framework that integrates formal verification and neurosymbolic reasoning to ensure these agents are both safe and interpretable during complex environmental missions.

Methodology

The proposed framework employs a multi-layered approach to address the inherent risks of RL in real-world deployments:

1. Online Safety Property Collection: This mechanism identifies "retrain areas" within the state space where the agent is most vulnerable to failure. By prioritizing these scenarios during training, the agent effectively learns from its mistakes.
2. Neurosymbolic (NeSy) RL: To accelerate learning, the system incorporates background knowledge via logical rules. This "soft probabilistic bias" guides exploration, preventing the agent from testing nonsensical actions.
3. Verification-guided Control Barrier Functions (CBFs): Acting as a "safety shield," this layer uses formal verification to define safe operating regions. If the RL policy proposes a dangerous action, a quadratic programming-based override ensures the drone remains within a mathematically guaranteed "safe set."
4. Gaussian Processes (GPs): To mitigate partial observability, GPs are used to construct a "belief state" model, allowing the drone to make informed decisions even when sensor data is incomplete.

Expand

Source, p. 2

Experimental Evaluation

The framework was tested across several benchmarks, including mapless navigation in a realistic aquatic simulator and the classic DoorKey (Minigrid) environment. These tests evaluated the agents' ability to handle logical dependencies and navigate without pre-existing maps. Furthermore, the researchers validated the generalizability of their safety-oriented methods on locomotion and power network tasks, culminating in a physical real-world deployment for environmental monitoring.

Key Results and Discussion

The integration of Control Barrier Functions proved highly effective. In aquatic navigation tests, the "PPO+CBF" and "PPOLag+CBF" models achieved a 0.0% collision rate, a stark contrast to the 11.22% rate observed in standard PPO baselines. Beyond safety, the "safety-wrapped" agents demonstrated superior mission effectiveness, with the PPO_penalty+CBF model reaching a 96.7% success rate.

Expand

Source, p. 2

The Neurosymbolic approach also addressed the "black box" nature of RL. By providing a "logical nudge," agents solved complex tasks that traditional reward-shaping methods failed to master. Additionally, the Gaussian Process-based model successfully recovered approximately 70% of the performance gap caused by partial observability.

Conclusion and Future Work

This research demonstrates that safety in RL does not require sacrificing performance. By applying mathematical filters at the output stage and injecting logic into the learning process, autonomous systems can become reliable tools for environmental science. Future efforts will focus on enhancing human-AI interaction to improve explainability for operators and conducting more rigorous field testing in diverse and unpredictable real-world aquatic environments.