Improving Autonomous Separation Assurance through Distributed Reinforcement Learning with Attention Networks

Marc W. Brittain, Luis E. Alvarez, Kara BreedenAAAI 2024 · DOI

As Advanced Air Mobility (AAM) moves toward reality, fleets of autonomous air taxis and cargo drones will soon populate urban skies. Traditional air traffic control, which relies on human intervention and centralized systems, cannot scale to the high-density "air corridors" envisioned for major metropolitan areas. This paper, presented at AAAI-24 by researchers from MIT Lincoln Laboratory, introduces a scalable, decentralized reinforcement learning (RL) framework designed to maintain safe separation between autonomous aircraft in these complex environments.

The Challenge of Separation Assurance

Ensuring every aircraft maintains a safe distance from others—known as separation assurance—presents several technical hurdles:

  1. Variable Density: Aircraft encounter a fluctuating number of neighbors, requiring neural networks that can handle variable-length inputs rather than fixed-length vectors.
  2. Environmental Uncertainty: Real-world surveillance data (e.g., ADS-B) is often noisy, and communication links between aircraft can be unreliable or fail entirely.
  3. Computational Scaling: Training agents for safety-critical flight requires massive amounts of data, often bottlenecked by synchronized training architectures that force actors to wait for learner updates.

Methodology: Attention and Asynchronous Learning

The researchers addressed these challenges by formulating the problem as a Markov Decision Process (MDP) and solving it using a novel extension of the Discrete Soft Actor-Critic (SACD) algorithm.

The distributed RL architecture decouples data collection from training, enabling a 10x increase in throughput.

Key components of the framework include:

  • Attention Networks: The model employs a multiplicative attention mechanism to process information from a variable number of "intruder" aircraft. This allows the agent to dynamically focus on the most immediate threats regardless of the total number of planes nearby.
  • Asynchronous Architecture: By utilizing an off-policy approach, the framework decouples data collection from the learning process. This allows 40 parallel actors to continuously feed experience into a centralized replay memory, resulting in a 10x increase in training transitions (2.36 billion vs. 256 million for previous on-policy methods).
  • Expanded Action Set: Beyond simple horizontal speed adjustments, the model incorporates vertical maneuvers (climbing and descending between altitude lanes), providing a richer set of conflict resolution strategies.

Experimental Evaluation

The study utilized the AAM-Gym testbed to simulate 20 days of representative traffic in New York City. The simulation environment featured 29 vertiports and a network of vertically stacked lanes at altitudes between 400 and 1,600 feet. Aircraft dynamics were modeled after the Eurocopter EC-135 using the OpenAP performance model.

Key Results and Performance

The proposed SACD-A model demonstrated superior safety and scalability compared to the previous state-of-the-art baseline, D2MAV-A.

Risk Ratio performance as a function of fleet size, illustrating the scalability and noise-robustness of the SACD-A model.
  • Safety and Scalability: As shown in the results, SACD-A consistently maintained a lower risk ratio than the baseline. Remarkably, SACD-A operating with noisy ADS-B surveillance outperformed the baseline even when the latter had access to perfect data. The model maintained a steady-state risk ratio even as fleet sizes scaled to 500 aircraft.
  • Operational Efficiency: The agent exhibited stable behavior, maintaining current speed and altitude approximately 91% of the time and only maneuvering when a genuine conflict was detected.
  • Robustness: The model remained effective even with low communication probabilities, suggesting it learned to anticipate the movements of non-cooperative aircraft through surveillance alone.

Conclusion and Future Work

By combining attention mechanisms with a high-throughput asynchronous RL architecture, this research provides a viable path toward safe, autonomous air traffic management. While the results are promising, the authors note that future work must address real-world sensor latency, complex weather patterns, and interoperability with existing collision avoidance systems like ACAS X before full-scale deployment is possible.

Summary updated

BibTeX
@inproceedings{brittain2024improving,
  author = {Marc W. Brittain and Luis E. Alvarez and Kara Breeden},
  title = {{Improving Autonomous Separation Assurance through Distributed Reinforcement Learning with Attention Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30321},
}
Figure