CoPeD-Advancing Multi-Robot Collaborative Perception: A Comprehensive Dataset in Real-World Environments

Yang Zhou, Long Quang, Carlos Nieto-Granda, Giuseppe Loianno

Abstract

Collaborative perception allows multi-robot teams to share sensor data to overcome individual limitations such as occlusions, sensor noise, and limited fields of view. However, existing datasets often prioritize SLAM-based coverage over perception-focused overlap or rely heavily on simulation. This paper introduces CoPeD (Collaborative Perception Dataset), a real-world, multi-modal dataset featuring heterogeneous air-ground robot teams. CoPeD provides overlapping views and automated semantic annotations, enabling research into robust collective understanding of complex environments.

Introduction

Robotic perception has advanced significantly in recent years, but most progress has occurred from the perspective of a single agent. Transitioning to multi-robot teams introduces significant hurdles, including the need to fuse data that is compressed, heterogeneous, and asynchronous.

Current datasets often suffer from a "SLAM bias," where the goal is to minimize overlap to maximize mapping efficiency. In contrast, collaborative perception thrives on overlapping views, allowing robots to "help" each teammates see through noise and occlusions. Furthermore, many existing collaborative datasets are purely simulated, failing to capture the real-world nuances of sensor jitter, lighting changes, and communication delays found in the wild.

Methodology and System Design

The researchers deployed a diverse team comprising two heavy-duty ground robots (Clearpath Warthog and Jackal) and two agile aerial drones (NYU ARPL Race robots). This heterogeneous team was equipped with a comprehensive sensor suite, including 3D LiDAR, IMUs, GNSS, RGBD cameras, and high-resolution stereo cameras.

The CoPeD framework: featuring heterogeneous air-ground robots and multi-modal data streams including thermal, RGB, aerial, and LiDAR views.

To ensure precise spatial alignment between agents, the team employed a combination of GPS, Visual-Inertial Odometry (VIO), and visual fiducial markers (AprilTags). A key innovation in the dataset's creation was the use of Foundation Models—such as SAM and Grounding DINO—to generate "zero-shot" semantic segmentation masks and depth maps, which were then temporally propagated for consistency across video frames.

Dataset Characteristics

CoPeD includes sequences from both indoor and outdoor environments, designed to test the limits of collaborative algorithms:

  • Indoor Sequences: Captured in a 38m x 60m laboratory space with complex occlusions from tables and chairs.
  • Outdoor Sequences: Captured in forested and residential-style environments featuring varied terrain, trees, and rocks.

Compared to 11 existing datasets (such as Lamp 2.0 and Opv2v), CoPeD is unique in offering real-world, air-ground, multi-modal data specifically for perception rather than just SLAM.

Collaborative Perception Insights

The dataset highlights the value of "perception compensation." For instance, an aerial robot's "bird's eye" view can provide critical context when a ground robot's sensors are obscured by vegetation. This synergy between heavy-duty ground sensors (like 64-beam LiDAR) and agile aerial perspectives is fundamental to robust multi-agent systems.

Experimental Validation and Results

The dataset was validated using Graph Neural Networks (GNNs) for tasks such as collaborative depth estimation and semantic robustness. Results indicated that multi-robot systems could effectively recover corrupted sensor data by communicating feature maps between teammates.

LiDAR point cloud processing within the CoPeD dataset, demonstrating object detection capabilities in complex environments.

Collaborative perception significantly improved the detection of distant objects compared to single-robot baselines. The dataset successfully captures "disturbances in the wild," such as GPS drift in forests and IMU inconsistencies, providing a rigorous benchmark for future algorithms.

Conclusion and Future Work

CoPeD represents a significant step toward bridging the gap between simulated and real-world multi-robot perception. While the current semantic annotations are primarily 2D, the authors plan to enrich the dataset with comprehensive 3D semantic labels and expand into even more diverse environmental contexts. By providing a real-world "playground" for multi-agent systems, CoPeD aims to foster a virtuous cycle between real-world data and high-fidelity robotic simulators.

Summary updated

BibTeX
@article{zhou2024coped,
  author = {Yang Zhou and Long Quang and Carlos Nieto-Granda and Giuseppe Loianno},
  title = {{CoPeD-Advancing Multi-Robot Collaborative Perception: A Comprehensive Dataset in Real-World Environments}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=mHf7SLKQKN},
}
Figure