Transformer-Based Reinforcement Learning for Multi-Robot Autonomous Exploration

Qihong Chen, Rui Wang, Ming Lyu, Jie Zhang2024 · DOI

TL;DR

This paper introduces a novel Transformer-based deep reinforcement learning (DRL) framework, TBDE-Net, for multi-robot autonomous exploration. By leveraging the Transformer's ability to process sequential data and capture long-range dependencies, TBDE-Net significantly improves exploration efficiency and reduces redundant scanning in unknown environments compared to existing methods.

What Problem Does This Paper Address?

Efficiently mapping unknown environments is crucial for applications like search and rescue. Traditional multi-robot exploration methods often struggle with the "repetitive exploration problem," where robots waste time re-exploring areas already covered by teammates. This inefficiency stems from limitations in how robots perceive and coordinate with each other, especially in decentralized systems where communication is limited. Existing approaches, like nearest frontier or utility-based methods, can become computationally complex or lead to suboptimal exploration patterns. This paper aims to overcome these limitations by developing a more intelligent and coordinated exploration strategy for multi-robot teams.

Methods

The core of this work is the Transformer-Based Decentralized Exploration Network (TBDE-Net). This framework builds upon the Multi-Agent Deep Reinforcement Learning (MADRL) paradigm, specifically using the MADDPG algorithm for centralized training and decentralized execution.

Here's a breakdown of the key methodological components:

  • Transformer as a Feature Extractor: The paper utilizes a Transformer encoder as a powerful feature extractor for the robot's local map and observations. Transformers excel at capturing long-range dependencies and contextual information, which is vital for understanding the environment and inferring teammate intentions, even with limited communication.
  • Decentralized Partially Observable Markov Decision Process (Dec-POMDP): The multi-robot exploration problem is formally modeled as a Dec-POMDP, a standard framework for multi-agent systems with partial observability.
  • Centralized Training, Decentralized Execution: The training process is centralized, allowing the system to learn optimal policies by observing all agents' states and actions. However, during execution, each robot acts independently based on its local observations and the learned policy, making the system robust to communication failures.
  • Policy and Critic Networks: TBDE-Net employs distinct policy and critic networks for each robot.
  • The policy network takes local observations (partial map, robot position, frontier points, teammate information) and outputs a probability distribution over potential exploration target points (frontier points). It uses a Transformer encoder for feature extraction, followed by an MLP and a policy head.
  • The critic network takes global observations (including action probabilities from the policy network) to evaluate the goodness of actions, guiding the learning process.
  • Novel Reward Function: A custom reward function is designed to encourage exploration of new areas, penalize repeated exploration, and account for time and decision-making costs.
  • Frontier Point Extraction: A machine vision-based approach is used to extract frontier points from local maps, which serve as candidate exploration targets.

Datasets

The paper does not use pre-existing datasets. Instead, it generates its own training and testing environments:

  • Training Environments: Randomly generated circular environments (222x222 pixels) with ten randomly placed obstacles (circles and squares of varying sizes) were used for training.
  • Simulation Environment: A Vrep simulation environment was used for testing, with a larger area (22.2m x 22.2m) and robots equipped with LiDAR sensors. This environment was designed to be more realistic, considering robot dynamics like inertia and acceleration.

Key Insights and Ideas

  • Transformer's Superiority in Contextual Understanding: The Transformer's ability to process sequential data and capture long-range dependencies proves highly effective in multi-robot exploration. It allows robots to better infer teammate intentions and avoid redundant exploration, even with limited communication.
  • Bridging the Gap Between Centralized Learning and Decentralized Execution: The MADRL approach, particularly with CTDE, effectively trains sophisticated policies in a centralized manner while allowing for robust decentralized operation, which is crucial for real-world robotic systems.
  • Importance of a Well-Designed Reward Function: The proposed reward function, which balances exploration, repetition penalties, and time costs, plays a significant role in shaping the robots' behavior towards efficient and comprehensive mapping.
  • Generalization to Realistic Scenarios: The method demonstrates good generalization capabilities, performing well in a simulated environment that includes robot dynamics, suggesting its potential for real-world deployment.

Key Results

  • Improved Exploration Efficiency: TBDE-Net consistently outperformed traditional methods (Nearest Frontier, Next-Best-View) and another DRL-based method (DME-DRL) in terms of both exploration time and distance traveled across various robot team sizes (2, 3, and 4 robots).
  • Reduced Redundancy: The Transformer-based approach significantly reduced redundant exploration, leading to faster and more complete map coverage.
  • Robustness to Communication Distance: The proposed method, along with planning-based and DME-DRL, showed insensitivity to variations in communication distance, indicating effective handling of intermittent communication.
  • Performance in Realistic Simulation: In a Vrep simulation considering robot dynamics, TBDE-Net achieved better results than the Nearest Frontier baseline, demonstrating its applicability in more complex scenarios. For instance, with 4 robots, TBDE-Net took 23.5s and 107.1m, while Nearest Frontier took 27.2s and 127.3m.

Limitations and Future Work

  • Fixed Input Dimension: The authors acknowledge that the fixed input dimension of the neural network might limit its applicability to environments of different sizes. Retraining would be necessary for new environment sizes.
  • Robot Team Size Variability: The current architecture requires retraining when the robot team size changes.
  • Future Work:
    • Develop policy networks that share parameters across robots to handle varying team sizes without retraining.
  • Explore feature pyramid structures to adapt the network to environments of different scales.
  • Investigate methods to make the network more adaptable to different robot team sizes.

Summary updated

BibTeX
@misc{chen2024transformer,
  author = {Qihong Chen and Rui Wang and Ming Lyu and Jie Zhang},
  title = {{Transformer-Based Reinforcement Learning for Multi-Robot Autonomous Exploration}},
  year = {2024},
  doi = {10.3390/s24165083},
}
Figure