Autonomous and Adaptive Role Selection for Multi-robot Collaborative Area Search Based on Deep Reinforcement Learning

Lina Zhu, Jiyu Cheng, Hao Zhang, Zhichao Cui, Wei Zhang, Yuehu LiuIEEE Transactions on Automation Science and Engineering 2023 · DOI · OpenReview

TL;DR

This paper introduces a novel hierarchical deep reinforcement learning approach for multi-robot area search that decouples task planning (role selection) from task execution. By enabling robots to dynamically choose between exploring new areas and covering known targets, their method achieves superior scalability and generalization in complex environments compared to existing state-of-the-art approaches.

What Problem Does This Paper Address?

Coordinated area search with multiple robots is crucial for applications like disaster response and exploration. The core challenge lies in efficiently balancing two simultaneous objectives: exploration (mapping unknown areas and discovering targets) and coverage (systematically searching known areas to locate and potentially "rescue" targets). Traditional methods often tackle these as separate, sequential tasks, leading to suboptimal performance and increased computational cost. Furthermore, existing approaches struggle with scalability to a large number of robots and varying environmental complexities. This paper aims to unify exploration and coverage into a single, efficient framework that can adapt to dynamic environments and a growing number of robots.

Methods

The authors propose a hierarchical multi-agent reinforcement learning (MARL) framework that separates task planning from task execution.

  • Hierarchical Structure: The system is divided into two main components:
  • Role Selection Module (Task Planning): This upper-level module uses a deep reinforcement learning (DRL) approach to assign roles to each robot. Robots learn to select between two primary roles: "explore" (seeking frontiers) and "cover" (seeking targets). This module is trained using a centralized critic and decentralized actors (MAPPO algorithm).
  • Primitive Policy Module (Task Execution): This lower-level module takes the assigned role as input and learns how to execute the corresponding sub-task. It uses local observations and the assigned role to generate primitive actions (move, turn, stop). This module is also trained using MAPPO.
  • Role Switching Mechanism: An "intelligent role switching mechanism" allows robots to dynamically change roles between timesteps, promoting a fluid interplay between exploration and coverage.
  • Goal-Augmented POMDP (GA-POMDP): The problem is formulated as a decentralized Partially Observable Markov Decision Process (POMDP) enhanced with sub-goals, where each robot receives a local observation augmented by a sub-goal.
  • Network Architecture: Convolutional Neural Networks (CNNs) are used to encode local observations, and these features, along with joint observations and role actions, are fed into Multi-layer Perceptrons (MLPs) for both role and primitive policy learning.

Datasets

The paper does not introduce new datasets. Instead, it uses a simulated environment generated by the gym simulation toolkit. The environments are described as randomly generated grid maps of varying sizes (25x25) with obstacles and target cells. The complexity is varied by altering the number of robots, obstacles, and target cells.

Key Insights and Ideas

  • Decoupling for Scalability: Separating task planning (role selection) from task execution is key to achieving scalability and adaptability in complex multi-robot scenarios.
  • Role-Based Adaptation: Assigning distinct roles (explore/cover) allows robots to specialize and optimize their actions based on the current environmental state and task objectives.
  • Dynamic Role Switching: The ability for robots to switch roles dynamically is crucial for efficiently balancing exploration and coverage, preventing robots from getting stuck in one mode.
  • Unified Exploration and Coverage: Integrating exploration and coverage into a single framework, rather than treating them sequentially, leads to more efficient and effective area search.
  • Role Weighting for Control: The authors demonstrate that adjusting the weights for exploration and coverage rewards allows for fine-tuning the robot's behavior, prioritizing one task over the other as needed.

Key Results

The proposed method demonstrates superior performance across various metrics and scenarios:

  • Generalization and Scalability: The method shows strong generalization across scenes of varying complexity (easy, medium, hard, super hard) and scalability with an increasing number of robots (4, 8, 15).
  • High Exploration and Coverage Percentages: In most scenarios, especially with more complex environments and a larger number of robots, the proposed method achieves significantly higher exploration and coverage percentages compared to baseline methods like Random, Greedy, GNN, and H2GNN. For instance, with 15 robots in the Super hard scene, the proposed method achieves 100% exploration and 91.5% coverage, outperforming ECC (100% exploration, 88.3% coverage).
  • Efficiency: The method demonstrates improved exploration efficiency, reaching 90% exploration faster than the ECC baseline, especially with fewer robots.
  • Robustness to Obstacles: In environments with a higher number of obstacles, the proposed method, along with ECC and Greedy, shows better performance in coverage percentage, indicating effective obstacle avoidance and focus on relevant areas.

Limitations and Future Work

  • Heuristic Role Weighting: The current method relies on manually setting the weight coefficients for exploration and coverage rewards. The authors acknowledge this limitation and suggest future work on dynamically adjusting these weights based on environmental states.
  • Environmental Representation: The use of 2D grid maps and CNNs for encoding observations might limit the representation of complex environments. Future work could involve enriching visual perception with 3D environment algorithms.
  • Complete Coverage Not Guaranteed: While the method achieves high coverage percentages, it does not guarantee complete coverage in all scenarios.
  • Future Directions: The authors plan to focus on intelligent methods for dynamically adjusting role weights and enhancing visual perception for more complex environments.

Summary updated

Abstract

In the tasks of multi-robot collaborative area search, we propose the unified approach for simultaneous mapping for sensing more targets (exploration) while searching and locating the targets (coverage). Specifically, we implement a hierarchical multi-agent reinforcement learning algorithm to decouple task planning from task execution. The role concept is integrated into the upper-level task planning for role selection, which enables robots to learn the role based on the state status from the upper-view. Besides, an intelligent role switching mechanism enables the role selection module to function between two timesteps, promoting both exploration and coverage interchangeably. Then the primitive policy learns how to plan based on their assigned roles and local observation for sub-task execution. The well-designed experiments show the scalability and generalization of our method compared with state-of-the-art approaches in the scenes with varying complexity and number of robots.
BibTeX
@article{zhu2023autonomous,
  author = {Lina Zhu and Jiyu Cheng and Hao Zhang and Zhichao Cui and Wei Zhang and Yuehu Liu},
  title = {{Autonomous and Adaptive Role Selection for Multi-robot Collaborative Area Search Based on Deep Reinforcement Learning}},
  journal = {IEEE Transactions on Automation Science and Engineering},
  year = {2023},
  doi = {10.1109/tase.2025.3579736},
}
Figure