Risk‐aware autonomous search and rescue with multiagent reinforcement learning

Aowabin Rahman, Salman Shuvo, Samrat Chatterjee, Mahantesh Halappanavar, Terje AvenRisk 2025 · DOI

TL;DR

This paper introduces a novel risk-aware multi-agent reinforcement learning (MARL) approach for autonomous search and rescue (SAR) missions. It tackles the challenge of unpredictable environments and adversarial agents by developing a system that learns to navigate and find targets while mitigating combined risks, achieving improved success rates in complex scenarios.

What Problem Does This Paper Address?

Autonomous multi-agent systems (MAS) are increasingly being explored for critical tasks like search and rescue (SAR) in dynamic and high-consequence environments. These missions require robots to locate targets (e.g., missing persons) efficiently, often in unknown or uncertain terrains. However, real-world SAR operations face significant challenges:

  • Environmental Uncertainty: The environment itself can be unpredictable, with unexpected obstacles or hazards (like pitfalls) emerging.
  • Adversarial Risks: Some agents within the MAS might exhibit malicious behavior, intentionally hindering the mission or providing false information. This is particularly concerning as autonomous agents might deviate from their learned behaviors during testing compared to training.
  • Coupled Risks: The interplay between environmental hazards and adversarial actions creates a complex risk landscape that current SAR systems struggle to handle effectively.

Existing approaches often focus on either environmental risks or adversarial risks, but not the combined effect. This paper aims to bridge this gap by developing a MARL framework that can robustly handle both simultaneously.

Methods

The core of this work lies in a risk-aware MARL algorithm built upon a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) formulation. Key methodological contributions include:

  • Risk Framework: The problem is framed using a risk triplet (events, consequences, uncertainties) and contemporary risk science knowledge, focusing on undesirable outcomes.
  • State Estimation with Local Awareness: Agents are equipped with enhanced state estimation capabilities that incorporate local awareness of neighboring regions. This includes information about obstacles, pitfalls, other agents, and retrieved targets. This local awareness allows agents to make decisions without explicit communication, enhancing robustness.
  • Composite Reward Structure: A novel reward function is designed to capture both environmental and adversarial risks. This structure incentivizes cooperative agents to explore efficiently while penalizing them for falling into pitfalls or being misled by adversaries. Separate reward functions are defined for cooperative and adversarial agents, reflecting their distinct objectives.
  • Centralized Training with Decentralized Execution (CTDE): The algorithm employs a CTDE framework. During training, a centralized critic has access to global information, enabling robust learning. However, during execution (testing), agents operate independently based on their local observations, making the system more practical and less prone to non-stationarity issues common in MAS.
  • Soft Actor-Critic (SAC): The underlying MARL algorithm utilizes a Soft Actor-Critic (SAC) approach, known for its ability to promote exploration and capture multiple modes of near-optimal solutions.

Datasets

Not discussed in the paper. The experiments were conducted in simulated gridworld environments created by the authors, with varying configurations of targets, obstacles, and pitfalls.

Key Insights and Ideas

  • The Synergy of Risks is Critical: The paper strongly suggests that ignoring the combined effects of environmental and adversarial risks leads to significantly worse performance in SAR missions. Training with both types of risks is crucial for robust real-world deployment.
  • Local Awareness is Key to Robustness: Equipping agents with local awareness of their surroundings, beyond just their immediate cell, allows them to navigate complex and uncertain environments more effectively without relying on constant communication.
  • Reward Shaping Matters: A carefully designed composite reward structure that penalizes undesirable outcomes (pitfalls, adversarial actions) and rewards exploration is essential for guiding agents towards optimal policies in high-risk scenarios.
  • CTDE Balances Learning and Execution: The CTDE paradigm proves effective by leveraging global information for robust training while enabling decentralized, practical execution in dynamic environments.

Key Results

  • Improved Success Rate: In the most complex scenario (Scenario 4, with both environmental and adversarial risks), the risk-aware MARL algorithm achieved a 57.14% success rate, significantly outperforming existing methods which achieved 37.50% (adversarial risks only) and 14.29% (environmental risks only). This is an improvement by a factor of over 1.5.
  • Reduced Retrieval Times (with caveats): While retrieval times generally increased with scenario complexity, the risk-aware MARL algorithm showed comparable or slightly better performance than baseline methods. Statistical significance tests indicated that the risk-aware MARL algorithm's retrieval times were significantly lower than MARL with environmental risks in Scenario 4.
  • Comparable Training Time: The risk-aware MARL algorithm required a higher training time (e.g., 60 hours for Scenario 4) compared to some baselines, but this trade-off was justified by the substantial improvement in success rate.
  • Generalizability: The learned policies demonstrated generalizability to new topographies without retraining, a key advantage for real-world deployment.

Limitations and Future Work

  • Simulation-Based: The current work is based on a gridworld simulation. While a "proof-of-concept," real-world deployment will require adapting state and action spaces for more complex environments (e.g., 3D).
  • Adversary Detection: The current framework assumes the presence of an adversary but doesn't explicitly focus on detecting or identifying adversarial agents.
  • Mixed Cooperative-Competitive Environments: Future work could explore adapting the algorithm for scenarios with mixed teams of cooperative and competitive agents.
  • Hierarchical Architectures: Integrating human operators into the decision-making loop via hierarchical architectures is a promising avenue.
  • Real-World High-Fidelity Simulation: Further research is planned for implementation in more realistic simulation environments like AirSim and Gazebo.
  • Communication Modeling: Investigating communication technologies for teams of unmanned vehicles and how risk-aware MARL can mitigate attacks like denial-of-service or man-in-the-middle is also a future direction.

Summary updated

Abstract

Autonomous navigation in dynamic high-consequence environments, such as search and rescue (SAR) missions, often relies on multiagent robotic systems that need to learn and adapt to changing conditions. Adversarial risks can introduce further challenges in such a setting where an autonomous agent may exhibit deviations in their learned actions from training to testing. Moreover, the uncertain environment itself may also evolve with additional obstacles that can emerge during testing compared to conditions when algorithmic training of autonomous agents was performed. In this paper, we first focus on mathematically formulating the autonomous SAR problem via a risk-aware multiagent reinforcement learning approach. Thereafter, we design and implement numerical experiments to evaluate our approach under diverse hazard scenarios with a centralized training and decentralized testing paradigm. Finally, we discuss our results and steps for further research.
BibTeX
@inproceedings{rahman2025riskaware,
  author = {Aowabin Rahman and Salman Shuvo and Samrat Chatterjee and Mahantesh Halappanavar and Terje Aven},
  title = {{Risk‐aware autonomous search and rescue with multiagent reinforcement learning}},
  booktitle = {Risk},
  year = {2025},
  doi = {10.1111/risa.70067},
}
Figure