Vision-based Distributed Multi-UAV Collision Avoidance via Deep Reinforcement Learning for Navigation

Huaxing Huang, Guijie Zhu, Zhun Fan, Hao Zhai, Yuwei Cai, Ze Shi, Zhaohui Dong, Zhifeng Hao

TL;DR

This paper introduces a novel vision-based, decentralized collision avoidance system for multi-UAVs using deep reinforcement learning. By training an autoencoder to extract features from depth images and combining it with a policy gradient method, each UAV can navigate complex 3D environments and avoid collisions without communication, demonstrating robust and scalable performance.

What Problem Does This Paper Address?

Navigating multiple unmanned aerial vehicles (UAVs) in a shared 3D space presents a significant challenge, especially when the density of UAVs increases. Traditional approaches often rely on centralized control, which can be computationally expensive and vulnerable to communication failures, or decentralized methods that still require inter-UAV communication. Existing vision-based methods may struggle with real-time performance due to complex processing pipelines. This paper tackles the problem of achieving robust, real-time, and fully decentralized collision avoidance for multi-UAV systems using only onboard sensors (depth cameras and inertial measurement units), without relying on external positioning or inter-UAV communication.

Methods

The core of the proposed solution is a deep reinforcement learning (DRL) framework that learns a decentralized collision-avoidance policy. Key aspects include:

  • Vision-based Input: The system uses depth images from a front-facing camera as a primary sensory input. To handle the high dimensionality of images, an autoencoder is employed.
  • Autoencoder for Feature Extraction: A deterministic autoencoder is trained jointly with the policy to extract latent representations from stacked depth images. This dimensionality reduction is crucial for efficient DRL training from pixel data.
  • Policy Gradient Reinforcement Learning: The paper utilizes a policy gradient-based DRL algorithm (specifically, an extension of Soft Actor-Critic with Autoencoder - SAC+AE) to learn the navigation policy.
  • Decentralized Execution: Each UAV independently executes the learned policy, making decisions based solely on its own observations (depth images, relative goal position, and current velocity).
  • Joint Training: The autoencoder (encoder and decoder) and the policy (actor and critic) are trained together. Notably, the gradients from the actor are prevented from updating the convolutional encoder to ensure stable training from image inputs.
  • Reward Function: A carefully designed reward function encourages reaching the goal (path efficiency) while penalizing collisions and proximity to obstacles (safety).

Datasets

The paper uses the Airsim simulator to generate simulated environments for training and evaluation. No specific public datasets are mentioned; rather, the authors create their own scenarios within the simulator.

Key Insights and Ideas

  • Vision is Sufficient: The research demonstrates that depth images, combined with inertial measurements, are sufficient for robust, decentralized multi-UAV collision avoidance, eliminating the need for explicit 3D spatial representations or communication.
  • Autoencoder for Efficiency: Using an autoencoder to learn latent representations from depth images significantly improves the efficiency and stability of DRL training compared to directly using raw pixel data.
  • Decentralization Without Communication: The success of the system in achieving collision-free navigation without any inter-UAV communication is a significant step towards more resilient and scalable multi-robot systems.
  • Emergent Swarm Behavior: The learned policies, even when decentralized, lead to visually regular and coordinated flight paths, suggesting the emergence of cooperative swarm behavior.

Key Results

  • High Success Rates: The proposed method consistently achieves high success rates across various scenarios (e.g., 99.4% in medium scenes, 95.2% in large scenes), outperforming baseline methods like RL+CNN and RL+VAE.
  • Improved SPL: The Success weighted by Path Length (SPL) metric also shows superior performance for the proposed method, indicating efficient navigation.
  • Robustness to Density: The system demonstrates excellent scalability and robustness, maintaining high performance even as the number of UAVs in the environment increases. For instance, in a circle scenario with up to 14 UAVs, the success rate remains high, while RL+VAE shows a significant performance degradation.
  • Faster Convergence: The learning curves in Figure 4 indicate that the proposed method converges to higher rewards during training compared to baselines.

Limitations and Future Work

  • Limited Obstacle Types: The current training scenarios do not include static or dynamic obstacles beyond other UAVs. The authors plan to incorporate these in future work.
  • Single Camera: The system relies on a single front-facing camera. The authors suggest that using multiple cameras could enhance environment perception and robustness in future extensions.
  • Extending to Real-World: While validated in simulation, extending the method to real-world scenes will require addressing challenges related to sensor noise, environmental variations, and ensuring overall robustness and safety.
  • Combining with MPC: The authors propose combining their learned policy with Model Predictive Control (MPC) for complex quadrotor control problems.
  • Reducing Extra Distance: While achieving high success rates, the method sometimes incurs a slightly higher "extra distance" compared to RL+VAE, which the authors aim to improve in future work while maintaining success rates.

Summary updated

Abstract

Online path planning for multiple unmanned aerial vehicle (multi-UAV) systems is considered a challenging task. It needs to ensure collision-free path planning in real-time, especially when the multi-UAV systems can become very crowded on certain occasions. In this paper, we presented a vision-based decentralized collision-avoidance policy learning method for multi-UAV systems. The policy takes depth images and inertial measurements as sensory inputs and outputs UAV's steering commands, and it is trained together with the latent representation of depth images using a policy gradient-based reinforcement learning algorithm and autoencoder in the multi-UAV three-dimensional workspaces. Each UAV follows the same trained policy and acts independently to reach the goal without colliding or communicating with other UAVs. We validate our method in various simulated scenarios. The experimental results show that our learned policy can guarantee fully autonomous collision-free navigation for multi-UAV in three-dimensional workspaces, and its navigation performance will not be greatly affected by the increase in the number of UAVs.
BibTeX
@inproceedings{huang2022vision,
  author = {Huaxing Huang and Guijie Zhu and Zhun Fan and Hao Zhai and Yuwei Cai and Ze Shi and Zhaohui Dong and Zhifeng Hao},
  title = {{Vision-based Distributed Multi-UAV Collision Avoidance via Deep Reinforcement Learning for Navigation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=U05vqqexHq},
}
Figure