Meta Reinforcement Learning Based Sensor Scanning in 3D Uncertain Environments for Heterogeneous Multi-Robot Systems

Junfeng Chen, Yuan Gao, Junjie Hu, Fuqin Deng, Tin Lun LamarXiv 2021 · arXiv

Introduction

Multi-robot sensor scanning is a critical capability for applications ranging from search-and-rescue to autonomous surveillance. The goal is to deploy a robotic team to maximize the observation coverage of a target area. However, real-world deployment faces significant challenges: the complexity of 3D environments, the inherent differences in robot capabilities (heterogeneity), and the unpredictability of environmental layouts. Traditional reinforcement learning (RL) approaches often struggle with "brittleness," where a model trained in one environment fails when the distribution of obstacles changes.

This paper presents meta-MRSS, a meta-reinforcement learning framework designed to enable heterogeneous teams of ground and aerial robots to adapt rapidly to novel 3D environments. By leveraging meta-learning, the system learns a generalized policy that can be fine-tuned to specific, unseen tasks with minimal additional training.

The meta-MRSS Framework

The proposed framework addresses the limitations of standard RL by combining heterogeneous agent modeling with a meta-learning architecture.

Heterogeneous Multi-Robot System

The system utilizes a mix of wheeled ground robots and quadcopters. These agents have fundamentally different constraints: wheeled robots are restricted to 2D movement but offer stability, while quadcopters provide 3D mobility and a wider sensor range (3.5m vs 2m). To manage this, meta-MRSS employs distinct policy networks ($\pi_w$ and $\pi_q$) tailored to each robot type's kinetic model.

The meta-MRSS architecture integrates heterogeneous policy networks for wheeled robots and quadcopters within a centralized training, decentralized execution framework.

Meta-Learning via MAML and POCA

At the heart of the system is the Model-Agnostic Meta-Learning (MAML) framework. MAML operates through a dual-loop optimization process. The "inner loop" adapts the policy to a specific environment configuration, while the "outer loop" updates a global "meta-policy" based on performance across a wide variety of tasks. This ensures the meta-policy serves as an optimal starting point for rapid adaptation.

For multi-agent coordination, the authors utilize the Post-Cognitive Actor-Critic (POCA) algorithm. POCA facilitates decentralized execution with centralized training, allowing individual robots to receive credit for actions that contribute to the collective team reward, even in complex 3D spaces.

Experimental Methodology

The researchers developed a high-fidelity simulation using the Unity ML-Agents toolkit. The training set consisted of 1,000 randomly generated 3D environments. These environments varied in the number, shape (cubes vs. irregular geometries), and placement of obstacles to simulate real-world uncertainty.

A representative 3D simulation environment featuring various obstacles used to evaluate the scanning success and adaptation speed of the multi-robot team.

Robots were equipped with raycast sensors to detect occupancy. The primary metric for success was the "scanning success rate," defined as the percentage of the target area successfully observed within a set time limit.

Key Results and Discussion

The meta-MRSS framework demonstrated significant improvements over baseline methods, including standard POCA, Domain Randomization, and traditional Transfer Learning.

  • Rapid Adaptation: meta-MRSS achieved convergence in approximately 4 million steps when introduced to new environments. In contrast, baseline methods required up to 14 million steps. This represents a 70-75% reduction in adaptation time, making the system viable for time-sensitive missions.
  • Superior Coverage: The proposed method reached an average scanning success rate of 87.12%. This outperformed standard Transfer Learning by 15% and standard POCA by 27%, highlighting the robustness of the meta-policy against environmental uncertainty.
  • Synergistic Heterogeneity: Ablation studies confirmed that the heterogeneous team (ground + aerial) was more efficient than homogeneous teams. The quadcopters provided high-level spatial data, while ground robots effectively filled coverage gaps, provided an asymmetric communication channel was established.

Conclusion

The meta-MRSS framework provides a robust solution for multi-robot sensor scanning in unpredictable 3D spaces. By "learning to learn," the system overcomes the brittleness of standard RL, allowing heterogeneous teams to deploy and adapt in minutes rather than hours. Future work will focus on bridging the sim-to-real gap by accounting for physical factors like wind and communication latency in real-world deployments.

Summary updated

Abstract

We study a novel problem that tackles learning based sensor scanning in 3D and uncertain environments with heterogeneous multi-robot systems. Our motivation is two-fold: first, 3D environments are complex, the use of heterogeneous multi-robot systems intuitively can facilitate sensor scanning by fully taking advantage of sensors with different capabilities. Second, in uncertain environments (e.g. rescue), time is of great significance. Since the learning process normally takes time to train and adapt to a new environment, we need to find an effective way to explore and adapt quickly. To this end, in this paper, we present a meta-learning approach to improve the exploration and adaptation capabilities. The experimental results demonstrate our method can outperform other methods by approximately 15%-27% on success rate and 70%-75% on adaptation speed.
BibTeX
@misc{chen2021meta,
  author = {Junfeng Chen and Yuan Gao and Junjie Hu and Fuqin Deng and Tin Lun Lam},
  title = {{Meta Reinforcement Learning Based Sensor Scanning in 3D Uncertain Environments for Heterogeneous Multi-Robot Systems}},
  year = {2021},
  eprint = {2109.13617},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.13617},
}
Figure