Distributed Non-Communicating Multi-Robot Collision Avoidance via Map-Based Deep Reinforcement Learning

Guangda Chen, Shunyi Yao, Jun Ma, Lifan Pan, Yu'an Chen, Pei Xu, Jianmin Ji, Xiaoping Chen

TL;DR

This paper presents a novel map-based deep reinforcement learning (DRL) approach for multi-robot collision avoidance in distributed, communication-free environments. By using egocentric local grid maps as input, the method achieves robust and efficient navigation, outperforming existing DRL-based approaches and demonstrating strong generalization capabilities.

What Problem Does This Paper Address?

Coordinating multiple robots to avoid collisions safely and efficiently, especially when they cannot communicate with each other and only have limited local sensing, is a significant challenge. Existing solutions often rely on robot-to-robot communication or require expensive, precise movement data (like velocities and paths) of other robots. These methods can be fragile, difficult to scale, and may not perform well in unstructured or dynamic environments. The paper aims to address this by developing a decentralized, communication-free collision avoidance system that is robust to noisy sensor data and does not require explicit inter-robot communication or movement data.

Methods

The core of the proposed solution is a map-based deep reinforcement learning (DRL) approach.

  • Egocentric Local Grid Maps: Instead of relying on raw sensor data or explicit robot states, the system uses egocentric local grid maps to represent the environment. These maps encode the robot's shape and the observable positions of nearby obstacles and other robots. This representation can be generated from various sensors (like 2D LiDAR or depth cameras) and sensor fusion.
  • Distributed Proximal Policy Optimization (DPPO): The authors adapt the DPPO algorithm for multi-robot collision avoidance. This allows robots to learn a shared policy in a distributed manner, where each robot collects its own experiences and contributes to updating the global policy. This approach combines centralized learning with decentralized execution.
  • Convolutional Neural Network (CNN) Policy: A CNN is trained to directly map three consecutive frames of egocentric local grid maps, along with the robot's relative local goal positions and orientation angles, to low-level control commands (linear and angular velocities).
  • Multi-Stage Curriculum Learning: To improve training efficiency and performance, a two-stage curriculum learning strategy is employed. The first stage trains the policy in simpler "random" environments, and the second stage refines it in more complex "circular" and "random" environments, with an increased number of robots.

Datasets

The paper does not explicitly mention using pre-existing datasets. Instead, it relies on:

  • Customized Simulator: A simulator built using OpenCV is used to generate training and testing environments. This simulator allows for the creation of various scenarios with static and dynamic obstacles, and different numbers and configurations of robots.
  • Real-World Experiments: The trained policy is deployed on four physical TurtleBot 2 robots equipped with LiDAR sensors.

Key Insights and Ideas

  • Map-based representation is powerful: Using egocentric local grid maps as input to the DRL agent proves to be more robust and informative than raw sensor data or explicit state information, especially in cluttered and dynamic environments.
  • Decentralized learning for scalability: The DPPO approach allows for efficient training of a shared policy across multiple robots, making it scalable to systems with many robots without requiring explicit communication.
  • Curriculum learning accelerates convergence: The two-stage curriculum learning strategy significantly improves the training process, leading to higher expected returns and better collision avoidance performance.
  • Generalization to unseen scenarios: The learned policy demonstrates remarkable generalization capabilities, performing well in scenarios and with robot configurations not encountered during training.
  • Robustness to sensor noise: The map-based approach shows resilience to sensor noise, outperforming rule-based methods in such conditions.

Key Results

  • Outperforms existing DRL methods: The map-based policy achieves superior performance in terms of success rate, extra time, and average linear velocity compared to a sensor-level DRL approach. For instance, in the "Circle scenario (12 robots, radius 3.5 m)", the Map-based Stage-2 policy achieved a success rate of 1.000 with an average extra time of 2.6133, while the Sensor-level policy also had a 1.000 success rate but with a higher extra time of 3.7290.
  • Effective in various scenarios: The approach demonstrated successful collision avoidance in diverse simulation scenarios, including non-cooperative robots, large-scale environments (up to 200 robots), and heterogeneous robots (different shapes and sizes).
  • Successful real-world deployment: The trained policy was successfully deployed on physical robots, performing well in static and dynamic obstacle avoidance, multi-robot coordination, and even long-range navigation in complex environments.
  • Robustness to sensor noise: The success rate of the map-based policy remained high even with increased sensor noise, while other methods showed a more significant degradation.

Limitations and Future Work

  • Limitations:
    • The current simulator does not inherently support 3D information, requiring workarounds for depth camera integration.
  • The accuracy of real-world robot localization can affect the smoothness of recorded trajectories.
  • Future Work:
    • Explore multi-sensor fusion to further leverage the map-based method.
  • Investigate other positioning methods (e.g., UWB, visual SLAM) to address laser positioning errors in dense environments.
  • Deploy the system in more dynamic and crowded environments, such as canteens and vegetable markets.

Summary updated

BibTeX
@inproceedings{chen2020distributed,
  author = {Guangda Chen and Shunyi Yao and Jun Ma and Lifan Pan and Yu'an Chen and Pei Xu and Jianmin Ji and Xiaoping Chen},
  title = {{Distributed Non-Communicating Multi-Robot Collision Avoidance via Map-Based Deep Reinforcement Learning}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=K_4BD9TS_Co},
}
Figure