Multi-Agent Reinforcement Learning-Based UAV Pathfinding for Obstacle Avoidance in Stochastic Environment

Qizhen Wu, Lei Chen, Kexin Liu, Jinhu LvarXiv 2023 · DOI · arXiv

Abstract

Traditional methods plan feasible paths for multiple agents in the stochastic environment. However, the methods' iterations with the changes in the environment result in computation complexities, especially for the decentralized agents without a centralized planner. Although reinforcement learning provides a plausible solution because of the generalization for different environments, it struggles with enormous agent-environment interactions in training. Here, we propose a novel centralized training with decentralized execution method based on multi-agent reinforcement learning, which is improved based on the idea of model predictive control. In our approach, agents communicate only with the centralized planner to make decentralized decisions online in the stochastic environment. Furthermore, considering the communication constraint with the centralized planner, each agent plans feasible paths through the extended observation, which combines information on neighboring agents based on the distance-weighted mean field approach. Inspired by the rolling optimization approach of model predictive control, we conduct multi-step value convergence in multi-agent reinforcement learning to enhance the training efficiency, which reduces the expensive interactions in convergence. Experiment results in both comparison, ablation, and real-robot studies validate the effectiveness and generalization performance of our method.

Introduction

Navigating a swarm of Unmanned Aerial Vehicles (UAVs) through a stochastic environment—where hazardous zones appear and disappear unpredictably—is a complex challenge. Traditional pathfinding methods like RRT* or Ant Colony Optimization (ACO) often require computationally expensive re-planning as the environment changes. While Reinforcement Learning (RL) offers a promising alternative due to its generalization capabilities, it typically suffers from sample inefficiency and struggles with the "curse of dimensionality" as the number of agents increases. This paper proposes a novel framework that combines Multi-Agent Reinforcement Learning (MARL) with Model Predictive Control (MPC) principles to achieve efficient, decentralized pathfinding.

Methodology

The proposed approach utilizes a Centralized Training with Decentralized Execution (CTDE) framework. Two key innovations address the limitations of standard MARL in dynamic environments:

Distance-Weighted Mean Field

To maintain scalability, the authors employ a mean field approach where agents observe a weighted average of their neighbors' states. Unlike standard mean field methods, this approach weights neighbors by distance, prioritizing information from closer agents that pose a more immediate collision risk. This reduces the observation space while preserving critical local information.

MPC-Based Multi-Step Value Convergence

Inspired by the rolling optimization in MPC, the agents utilize a learned virtual environment model to perform multi-step "imagined" rollouts during training. This allows the agents to update their value functions based on predicted future states, significantly accelerating convergence and reducing the need for expensive real-world interactions. To ensure stability, an adaptive truncation mechanism adjusts the prediction horizon based on the current accuracy of the virtual environment model.

Illustration of the distance-weighted mean field approach and the multi-step value convergence mechanism.

Experimental Results and Validation

The framework, designated as CTFDE-MPC, was evaluated in a custom 3D multi-UAV simulation environment accounting for physical factors like gravity and drag.

Performance and Scalability

CTFDE-MPC demonstrated superior convergence speed and higher cumulative rewards compared to decentralized DDPG. Notably, the system exhibited strong zero-shot generalization; policies trained on 6 UAVs were successfully deployed on swarms of up to 12 UAVs without additional training. While traditional methods like ACO saw exponential increases in planning time as environmental stochasticity increased, the proposed method maintained a near-constant computation time (~1.05s to 1.68s).

Real-Robot Implementation

The method's practical utility was validated using Crazyflie 2.1 nano-quadrotors. The experimental setup integrated an Optitrack motion-capture system for localization and a ROS-based control system. The agents successfully navigated around dynamic hazardous areas to reach target zones, demonstrating a successful sim-to-real transition.

Experimental environment showing target and hazardous areas (left) and the overall system architecture (right).

Conclusion

This work presents a robust MARL framework for UAV swarm navigation in unpredictable environments. By integrating distance-weighted mean field observations and MPC-inspired value convergence, the proposed method achieves high training efficiency, real-time execution, and excellent scalability. Future work will focus on transitioning to fully onboard sensing and handling more complex dynamic obstacles.

Summary updated

Abstract

Traditional methods plan feasible paths for multiple agents in the stochastic environment. However, the methods' iterations with the changes in the environment result in computation complexities, especially for the decentralized agents without a centralized planner. Although reinforcement learning provides a plausible solution because of the generalization for different environments, it struggles with enormous agent-environment interactions in training. Here, we propose a novel centralized training with decentralized execution method based on multi-agent reinforcement learning, which is improved based on the idea of model predictive control. In our approach, agents communicate only with the centralized planner to make decentralized decisions online in the stochastic environment. Furthermore, considering the communication constraint with the centralized planner, each agent plans feasible paths through the extended observation, which combines information on neighboring agents based on the distance-weighted mean field approach. Inspired by the rolling optimization approach of model predictive control, we conduct multi-step value convergence in multi-agent reinforcement learning to enhance the training efficiency, which reduces the expensive interactions in convergence. Experiment results in both comparison, ablation, and real-robot studies validate the effectiveness and generalization performance of our method.
BibTeX
@misc{wu2023multi,
  author = {Qizhen Wu and Lei Chen and Kexin Liu and Jinhu Lv},
  title = {{Multi-Agent Reinforcement Learning-Based UAV Pathfinding for Obstacle Avoidance in Stochastic Environment}},
  year = {2023},
  eprint = {2310.16659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.16659},
}
Figure