Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning

Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, Jia Pan

TL;DR

This paper introduces a novel deep reinforcement learning framework for decentralized multi-robot collision avoidance that directly maps raw sensor data to robot actions. By training across diverse scenarios and stages, their approach achieves robust, efficient, and generalizable navigation, significantly closing the performance gap with centralized methods.

What Problem Does This Paper Address?

Developing effective collision avoidance for multiple robots operating in a decentralized manner is a significant challenge. In decentralized systems, each robot must make decisions based only on its local sensor information, without a global view of other robots' intentions or states. Existing decentralized methods often rely on extracting complex agent-level features or require perfect sensing and communication, which are not always practical or robust in real-world scenarios. These limitations lead to performance that is often inferior to centralized approaches, which, while effective, struggle to scale to large numbers of robots and are vulnerable to communication failures. The authors aim to bridge this gap by creating a decentralized policy that is sensor-level, robust to partial observability, and performs comparably to centralized systems.

Methods

The core of this work is a sensor-level collision avoidance policy implemented as a deep neural network. Unlike agent-level policies that require explicit state estimation of other robots, this policy directly takes raw sensor measurements (2D laser scans, relative goal position, and current velocity) as input and outputs steering commands (translational and rotational velocity).

To train this policy effectively, the authors propose a multi-scenario multi-stage training framework. This framework leverages a policy gradient-based reinforcement learning algorithm, specifically an adaptation of Proximal Policy Optimization (PPO). The training process involves:

  • Multi-scenario training: Robots are trained across a variety of simulated environments with different obstacle configurations and robot arrangements.
  • Multi-stage training: This is a curriculum learning approach. The first stage focuses on training with a smaller number of robots (20) in simpler, obstacle-free random scenarios to quickly learn basic collision avoidance. The second stage then scales up to more robots (58) and more complex scenarios, building upon the foundation from the first stage.
  • Decentralized execution, centralized training: While each robot executes its learned policy independently, the policy parameters are updated centrally using data collected from all robots simultaneously. This allows for efficient learning from a large amount of experience.

The neural network architecture consists of convolutional layers to process laser scans, followed by fully connected layers. The output is a mean and log standard deviation for a Gaussian distribution from which the action (velocity) is sampled, promoting exploration.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, it relies on simulated environments created using the Stage mobile robot simulator. These simulations serve as the training and evaluation grounds. The authors created a suite of diverse scenarios, including:

  • Circle scenarios: Robots arranged in a circle aiming for antipodal positions.
  • Random scenarios: Robots and obstacles placed randomly.
  • Corridor scenarios: Robots navigating through narrow passages.
  • Group scenarios: Testing cooperation in group swaps, crossings, and corridor movements.
  • Heterogeneous and non-cooperative scenarios: For generalization testing.

These simulated environments are designed to be rich and complex, covering various interaction situations and obstacle types to ensure robust policy learning.

Key Insights and Ideas

  • Sensor-level policies are viable and robust: Directly mapping raw sensor data to actions bypasses the need for complex state estimation and communication, making the policy more practical and less prone to perception errors.
  • Multi-scenario, multi-stage training is crucial for generalization: Training across a wide range of environments and using a curriculum learning approach significantly improves the policy's ability to generalize to unseen situations, including heterogeneous robot shapes and non-cooperative agents.
  • Deep RL can close the performance gap: The proposed framework demonstrates that deep reinforcement learning can achieve performance comparable to, and in some metrics better than, established methods like NH-ORCA, even in complex, large-scale scenarios.
  • Decentralized execution with centralized training is an effective paradigm: This approach allows for efficient learning from massive amounts of data while maintaining the benefits of decentralized operation.

Key Results

The learned policy demonstrated strong performance across various metrics:

  • High Success Rates: Achieved near-perfect success rates (1.0) in many circle and random scenarios, outperforming NH-ORCA and supervised learning (SL) policies.
  • Reduced Extra Time and Distance: Significantly lower extra time and distance metrics compared to NH-ORCA and SL policies, indicating more efficient and direct paths. For instance, in circle scenarios with 20 robots, the extra time was 0.271/0.005s compared to NH-ORCA's 1.750/0.654s.
  • Improved Average Speed: The learned policy achieved higher average speeds, contributing to faster navigation.
  • Excellent Generalization: The policy successfully navigated heterogeneous groups of robots with different sizes and shapes, and scaled to a large environment with 100 robots without any fine-tuning. It also generalized to avoid non-cooperative agents.
  • Robustness in Complex Scenarios: Outperformed NH-ORCA in group crossing and group swap scenarios, suggesting better cooperative behavior learning.

Limitations and Future Work

  • Limitations:
    • The current work focuses on simulated environments. While the generalization is promising, real-world validation is still needed.
  • The learned policy is primarily for local collision avoidance and cannot replace a global path planner for complex tasks involving dense obstacles or intricate scheduling.
  • The performance metrics (extra time, extra distance) are measured over all robots, which might mask individual robot performance variations.
  • Future Work:
    • Extending the framework to real-world robot systems.
  • Integrating the learned local collision avoidance policy with a global path planning module for more complex navigation tasks.
  • Investigating the policy's performance in scenarios with more dynamic and unpredictable environments or agents.
  • Further exploring the impact of different reward functions and training curricula.

Summary updated

Abstract

Developing a safe and efficient collision avoidance policy for multiple robots is challenging in the decentralized scenarios where each robot generates its paths without observing other robots' states and intents. While other distributed multi-robot collision avoidance systems exist, they often require extracting agent-level features to plan a local collision-free action, which can be computationally prohibitive and not robust. More importantly, in practice the performance of these methods are much lower than their centralized counterparts. We present a decentralized sensor-level collision avoidance policy for multi-robot systems, which directly maps raw sensor measurements to an agent's steering commands in terms of movement velocity. As a first step toward reducing the performance gap between decentralized and centralized methods, we present a multi-scenario multi-stage training framework to learn an optimal policy. The policy is trained over a large number of robots on rich, complex environments simultaneously using a policy gradient based reinforcement learning algorithm. We validate the learned sensor-level collision avoidance policy in a variety of simulated scenarios with thorough performance evaluations and show that the final learned policy is able to find time efficient, collision-free paths for a large-scale robot system. We also demonstrate that the learned policy can be well generalized to new scenarios that do not appear in the entire training period, including navigating a heterogeneous group of robots and a large-scale scenario with 100 robots. Videos are available at https://sites.google.com/view/drlmaca.
BibTeX
@inproceedings{long2018towards,
  author = {Pinxin Long and Tingxiang Fan and Xinyi Liao and Wenxi Liu and Hao Zhang and Jia Pan},
  title = {{Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=qpXD3Y3Ril},
}
Figure