Multi-UAV Formation Control with Static and Dynamic Obstacle Avoidance via Reinforcement Learning

Yuqing Xie, Chao Yu, Hongzhi Zang, Feng Gao, Wenhao Tang, Jingyi Huang, Jiayu Chen, Botian Xu, Yi Wu, Yu WangIROS 2025 · DOI

TL;DR

This paper introduces a novel two-stage reinforcement learning (RL) approach for controlling multiple drones (UAVs) to maintain formation while simultaneously avoiding both static and dynamic obstacles. Their method achieves superior performance in simulation and real-world deployment compared to existing planning-based and RL-based methods, demonstrating robust scalability and generalization.

What Problem Does This Paper Address?

Coordinating multiple drones for tasks like search and rescue or payload delivery is incredibly complex. It requires not only individual drone capabilities like flight control and obstacle avoidance but also sophisticated coordination to maintain a desired formation. Adding dynamic obstacles (like birds or other moving objects) and the inherent "sim-to-real" gap (differences between simulated and real-world performance) significantly escalates this challenge. Existing methods often tackle formation control or obstacle avoidance, but rarely both, especially when dealing with dynamic obstacles and the need for robust, real-world deployable solutions.

Methods

The core of this paper is a two-stage reinforcement learning (RL) pipeline:

  1. Stage 1: Reward Scalarization (Weight Selection): Instead of directly training an RL agent for all objectives (directed flight, formation maintenance, obstacle avoidance, action smoothing), the authors first focus on finding the right balance between these objectives. They do this by searching for an optimal reward function in a simplified scenario. This involves randomly sampling weights for each objective and evaluating which combination yields the best "satisfaction rate" (how often all objectives are met). This stage essentially tunes the agent's priorities.
  2. Stage 2: Curriculum Learning and Attention-Based Encoder: Once the optimal reward function is identified, it's applied to a more complex task. To accelerate training and improve performance, they employ curriculum learning (CL). This means starting with simpler scenarios (e.g., obstacle-free flight) and gradually increasing the difficulty (static obstacles, then mixed static/dynamic obstacles). A key innovation here is an attention-based observation encoder. This module helps the agent process information about its surroundings, particularly the varying number of obstacles, and better understand relationships between drones, leading to improved formation maintenance and adaptability.

Datasets

The paper does not explicitly mention using pre-existing datasets for training or evaluation. Instead, they rely on extensive simulations using the OmniDrones simulator and conduct real-world experiments with Crazyflie 2.0 drones. The "dataset" can be considered the diverse scenarios they create within these environments, including varying numbers of static and dynamic obstacles.

Key Insights and Ideas

  • The Importance of Reward Shaping: Finding the right balance between competing objectives (formation vs. obstacle avoidance) is critical. A two-stage approach, where objective weights are optimized first, proves more effective than trying to learn everything at once.
  • Curriculum Learning is Key for Complexity: Tackling complex multi-agent, multi-objective tasks with RL is significantly accelerated and improved by gradually increasing the task difficulty.
  • Attention Mechanisms Enhance Scalability: The attention-based observation encoder is crucial for handling varying numbers of obstacles and improving inter-drone coordination, making the policy more robust and scalable.
  • Action Smoothing for Sim-to-Real: Including an "action smoothing" objective in the reward function is vital for bridging the sim-to-real gap, ensuring policies trained in simulation perform well on physical robots.

Key Results

  • Superior Performance: The proposed method outperforms state-of-the-art baselines (Swarm-Formation, R-Mader, Swarm-RL) in terms of both Collision-Free Rate (CFR) and Formation Maintenance (FM) across static, dynamic, and mixed obstacle scenarios in simulation.
  • Scalability: The method shows strong scalability, with only a slight degradation in performance as the number of obstacles increases, attributed to the attention mechanism.
  • Real-World Success: The learned policy was successfully deployed on physical Crazyflie 2.0 drones without additional fine-tuning, demonstrating excellent zero-shot Sim2Real transfer. It achieved higher CFR than baselines in real-world tests.
  • Flexibility: The reward scalarization stage allows for diverse behaviors, enabling users to select policies that best align with their specific preferences.

Limitations and Future Work

  • Communication Assumption: The current approach assumes perfect communication between drones.
  • Limited Sensing: The system does not currently incorporate vision-based sensing.
  • Structured Environments: The work is demonstrated in structured environments.

Future work suggested includes:

  • Developing a vision-based policy for more robust obstacle handling.
  • Making the system more robust to communication delays.
  • Extending the approach to handle unstructured environments.

Summary updated

BibTeX
@inproceedings{xie2025multi,
  author = {Yuqing Xie and Chao Yu and Hongzhi Zang and Feng Gao and Wenhao Tang and Jingyi Huang and Jiayu Chen and Botian Xu and Yi Wu and Yu Wang},
  title = {{Multi-UAV Formation Control with Static and Dynamic Obstacle Avoidance via Reinforcement Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246743},
}
Figure