Abstract
Mission Engineering (ME) for complex System of Systems (SoS) requires adaptive decision-making in highly dynamic and uncertain environments. Traditional static planning often fails to account for the evolving nature of missions like wildfire suppression. This paper proposes a novel framework that integrates high-fidelity digital twins with Reinforcement Learning (RL) to coordinate aerial firefighting assets. By training a Proximal Policy Optimization (PPO) agent within a physics-based simulation of the Palisades wildfire, the study demonstrates that intelligent mission coordinators can significantly outperform baseline tactical assignments. The results indicate that the RL agent not only improves mean mission effectiveness but also substantially reduces performance variability, effectively mitigating catastrophic mission failures.
Introduction
Traditional Systems Engineering (SE) excels at designing individual, monolithic systems. However, modern challenges increasingly demand a System of Systems (SoS) approach, where multiple independent assets—such as a fleet of aircraft and ground units—must collaborate to achieve a shared mission objective. Current SoS architectures are often "brittle"; because mission environments are dynamic, static plans formulated at the start of a mission frequently become obsolete as conditions shift.
Furthermore, the mathematical design space for these missions is vast, rendering traditional optimization tools too slow for real-time tactical adjustments. This research bridges the gap by utilizing Digital Engineering (DE) to create a high-fidelity simulation "sandbox" where an intelligent agent can learn robust, real-time tactical management through repeated interaction.
Methodology
The researchers developed a three-stage methodology to transition from a physical environment to an intelligent coordinator:
- Digital Mission Model: A physics-based grid representation of a geographical area, incorporating fuel types (e.g., pines, leaf litter), weather variables (wind, temperature, humidity), and critical infrastructure.
- Agent-Based Simulation: Utilizing a simulation environment from the German Aerospace Center (DLR), the model captures the complex interactions between firefighting assets and fire propagation.
- Reinforcement Learning (RL): The mission is formulated as a Markov Decision Process (MDP), with a centralized coordinator trained using Proximal Policy Optimization (PPO).
Instead of low-level flight control, the RL agent makes strategic decisions every 10 minutes by assigning tactical triplets: selecting a Point of Interest (POI), determining the tracking method (e.g., following the fire front), and choosing the suppression mode (direct vs. indirect). This abstraction allows the agent to focus on high-level strategy across 24 possible tactical combinations per aircraft.
Experimental Framework
The study utilizes a case study based on the Palisades wildfire in California. The simulation was calibrated to replicate the specific catastrophic conditions of that event, including temperature, humidity, and wind speed. The agent's state space is defined by a comprehensive set of environmental and operational variables, as detailed in the table below.
The performance is evaluated using a custom Measure of Effectiveness (MoE) function that aggregates four critical metrics: casualties, cost of burnt area, total area destroyed, and $CO_2$ emissions.
Key Results and Insights
The RL agent was trained over 3,000 simulated episodes, showing a clear performance gap over random baselines starting around the 1,200th iteration.
- Quantitative Success: The PPO agent achieved a mean MoE of 0.862, significantly higher than the random baseline of 0.755 ($p < 0.001$). In the final 1,000 episodes, the agent stabilized at a median MoE of 0.935.
- Stability and Reliability: A critical finding was the reduction in outcome variability. While random tactics occasionally yielded successful results by chance, the RL agent consistently avoided the "left-tail" anomalies—catastrophic mission failures—that plagued the baseline.
- The Power of the Sandbox: High-fidelity digital twins allowed the agent to "fail" thousands of times safely, learning which tactics succeed under complex weather combinations that are mathematically difficult to model with simple equations.
Limitations and Future Work
While promising, the current model assumes perfect information, where the agent has total visibility of the fire and weather. Real-world applications must contend with the "fog of war," including sensor errors and communication delays. Future research will explore heterogeneous fleets (e.g., combining tankers with drones) and focus on Explainable AI (XAI) to ensure human commanders can trust and interpret the agent's tactical recommendations. Additionally, moving beyond tactical triplets to more continuous control may uncover even more specialized maneuvers for fire suppression.
Summary updated