Dashing for the Golden Snitch: Multi-Drone Time-Optimal Motion Planning with Multi-Agent Reinforcement Learning

Xian Wang, Jin Zhou, Yuanli Feng, Jiahao Mei, Jiming Chen, Shuo Li

Introduction

Autonomous quadrotors have demonstrated remarkable agility in single-agent time-optimal flight. However, extending these capabilities to multi-drone swarms presents significant challenges. Traditional optimization-based methods struggle with the "curse of dimensionality," where computational complexity grows exponentially with the number of agents, making real-time onboard execution unfeasible. Learning-based approaches often prioritize safety over speed or fail to validate performance in high-speed, real-world environments. This work introduces a decentralized reinforcement learning framework that enables swarms to fly time-optimal trajectories while autonomously avoiding mid-air collisions.

Methodology

The problem is formulated as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP) and addressed using Multi-Agent Reinforcement Learning (MARL).

Centralized Training, Decentralized Execution (CTDE)

The framework employs a CTDE strategy. During the training phase, a centralized "critic" network has access to the global state of all drones to provide accurate value estimations and feedback. For execution, each drone utilizes a lightweight "actor" policy network ($\pi_\theta$) that relies solely on local observations: its own state, the next two waypoints, and the relative positions and velocities of neighboring drones.

The CTDE framework: Centralized training uses a shared rollout buffer and global state, while decentralized execution deploys individual policy networks on each drone.

Soft Collision-Free Mechanism

To balance speed and safety, the system incorporates a "soft" collision-free mechanism. Unlike standard RL approaches that terminate episodes upon collision, this method uses continuous penalties and safety margins inspired by classical optimization. This allows agents to explore the limits of safe flight without the disruptive effects of hard terminations, facilitating the learning of high-speed avoidance maneuvers.

Policy Optimization

The agents are trained using Independent Proximal Policy Optimization (IPPO) with two key enhancements: invalid-experience masking (to filter data from crashed drones) and value normalization to stabilize the training process. The resulting policy is computationally efficient, running at 100 Hz on onboard hardware to enable millisecond-level replanning.

Experimental Evaluation

The system was evaluated in a customized simulation environment based on gym-pybullet-drones and validated through real-world flight tests.

Simulation Benchmarks

Agents were tested on five distinct race tracks: Eight, Arrow, Star, Lotus, and Candy. Additionally, the system was benchmarked against the 2019 AlphaPilot Challenge course.

Trajectory and velocity profiles across five different race tracks, demonstrating the system's ability to maintain high speeds while following complex paths.

Real-World Validation

Physical experiments were conducted using 305g quadrotors in a 5.5 m × 5.5 m × 2.0 m flight arena. The simulation-trained policies were deployed directly onto the hardware without fine-tuning, demonstrating robust sim-to-real transfer.

Results and Discussion

The proposed framework achieved "champion-level" agility across various scenarios.

  • High-Speed Performance: In real-world tests, a two-drone swarm reached speeds of 13.65 m/s. In simulation, a five-drone swarm on the AlphaPilot course achieved peak speeds of 27.1 m/s with an 83.8% success rate.
  • Emergent Behavior: Drones exhibited sophisticated self-organization, such as transitioning from high-risk head-on approaches to safer same-direction alignments and employing "near-miss" strategies to preserve momentum.
  • Efficiency vs. Safety: A trade-off was observed where increasing the number of drones led to slightly longer lap times, as agents adopted wider trajectories to maintain safety margins.
  • Competitive Optimality: On the "Star" track, the RL policy was only 0.44s slower than the computationally expensive CPC (Constrained Policy Control) method, with multi-drone coordination adding minimal overhead.

Conclusion

This research demonstrates that decentralized MARL can effectively manage multi-drone swarms for time-optimal motion planning. By combining CTDE with soft safety constraints, the system achieves high-speed flight and robust collision avoidance on resource-constrained hardware. Future work will focus on incorporating temporal reasoning via LSTMs, integrating advanced sensing modalities like LiDAR for GPS-denied environments, and enhancing coordination through team-based reward structures.

Summary updated

BibTeX
@inproceedings{wang2025dashing,
  author = {Xian Wang and Jin Zhou and Yuanli Feng and Jiahao Mei and Jiming Chen and Shuo Li},
  title = {{Dashing for the Golden Snitch: Multi-Drone Time-Optimal Motion Planning with Multi-Agent Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2025},
  doi = {10.1109/icra55743.2025.11128442},
}
Figure