Semi-Markovian Planning to Coordinate Aerial and Maritime Medical Evacuation Platforms

Mahdi Al-Husseini, Kyle Hollins Wray, Mykel J. Kochenderfer

TL;DR

This paper introduces a novel approach to medical evacuation in maritime environments by using underway watercraft as dynamic exchange points between helicopters. By modeling this as a Semi-Markov Decision Process and employing Monte Carlo Tree Search, they significantly improve evacuation efficiency and reduce response times, demonstrating a real-world deployment with the US Army.

What Problem Does This Paper Address?

Coordinating medical evacuations in vast maritime environments presents a significant challenge. Traditional methods rely on either fast but range-limited aircraft or slow but long-range hospital ships. The problem intensifies when dealing with multiple islands and the need for flexible, rapid patient transfers. Existing approaches struggle to dynamically integrate various assets, especially underway watercraft whose trajectories are often dictated by non-medical missions. This paper tackles the problem of optimally selecting these moving watercraft as intermediary "ambulance exchange points" to bridge the gap between aerial and maritime capabilities, thereby expediting patient movement and extending evacuation reach.

Methods

The core of this work lies in formulating the problem as a Semi-Markov Decision Process (SMDP). This allows for modeling the complex dynamics of the evacuation environment, including aircraft and watercraft positions, velocities, and utilization histories, while accounting for the significant impact of sojourn times (time to resolve an evacuation request) on rewards.

To solve this SMDP and determine optimal exchange point selections and aircraft dispatch times, the authors employ Monte Carlo Tree Search (MCTS) with root parallelization. MCTS is an anytime algorithm well-suited for complex search spaces, and root parallelization helps to overcome the inherent uncertainty and sparsity of evacuation requests by exploring multiple search trees simultaneously. A key aspect is the use of a greedy policy for simulation roll-outs, which prioritizes actions that minimize transfer time based on immediate proximity.

The reward functions are designed to balance short-term and long-term rewards, incorporating both patient survivability (modeled with a Weibull survival function) and battlefield clearance considerations (a linear component).

Datasets

Not discussed in the paper. The authors describe a simulated environment and a real-world deployment exercise.

Key Insights and Ideas

  • Watercraft as Dynamic Exchange Points: The paper demonstrates that strategically using underway watercraft as flexible exchange points can significantly enhance medical evacuation capabilities, combining the speed of aircraft with the reach of maritime assets.
  • SMDP for Complex Coordination: Modeling the problem as an SMDP proves effective in capturing the intricate dependencies and time-varying nature of maritime evacuation coordination.
  • MCTS for Real-time Decision Making: MCTS, particularly with root parallelization, provides a robust framework for making optimal dispatch and exchange point decisions in a dynamic and uncertain environment.
  • Balancing Survivability and Battlefield Clearance: The fused reward function effectively balances the critical need to preserve patient life with the operational requirement to clear the battlefield.
  • Geographic Imbalances: The distribution of casualties across islands significantly impacts exchange point selection, highlighting the need for adaptive strategies.
  • Watercraft Density as a Substitute for Control: The study suggests that a sufficient density of participating watercraft can lead to minimized disparities in response times between different evacuation platoons, even without direct control over watercraft movement.

Key Results

  • An optimal policy utilizing watercraft exchange points outperforms an optimal policy without them and a greedy policy by 35% and 40%, respectively, in terms of total rewards.
  • Watercraft exchange points generally result in a 30-35% improvement in total rewards.
  • In a real-world deployment, the incident response time from evacuation request notification to role three patient delivery was 76 minutes, with only 16 minutes attributed to maritime traffic delays.
  • The simulation model accurately captured real-world dynamics, with deployment outcomes closely matching simulated evacuation times.
  • The paper led to proposed revisions to Army doctrinal publications on medical evacuation, establishing the watercraft exchange point as a standard technique.

Limitations and Future Work

  • Partial Observability: The current model assumes full observability of the environment. Future work could explore models that account for partial observability, especially in scenarios where communication fails.
  • Communication and Contingencies: The paper acknowledges that assuming full observability requires platforms to communicate intentions and state information. Developing models that handle communication failures and contingencies is a crucial next step.
  • Asynchronous Actions: The coordination of multiple platforms with asynchronous actions is susceptible to sudden changes. Future work should focus on how update infrastructure can gauge risk when determining whether to amend existing guidance.
  • Watercraft Control: While the study shows watercraft density can be effective, future research could investigate the benefits of incorporating explicit control over watercraft movement into the decision-making process.

Summary updated

BibTeX
@inproceedings{alhusseini2025semi,
  author = {Mahdi Al-Husseini and Kyle Hollins Wray and Mykel J. Kochenderfer},
  title = {{Semi-Markovian Planning to Coordinate Aerial and Maritime Medical Evacuation Platforms}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35138},
}
Figure