CoPeD-Advancing Multi-Robot Collaborative Perception: A Comprehensive Dataset in Real-World Environments

Yang Zhou, Long Quang, Carlos Nieto-Granda, Giuseppe Loianno

Robotic perception has come a long way in the last decade, but most of that progress has happened in a vacuum—specifically, the vacuum of a single robot’s perspective. When you move to a team of robots, the game changes. You have to fuse data that is often compressed, intermittent, heterogeneous (different types of sensors), and asynchronous.

A new paper from researchers at NYU and the U.S. Army Research Laboratory introduces CoPeD (Collaborative Perception Dataset), a comprehensive real-world dataset designed to move multi-robot systems past these hurdles.

TL;DR

CoPeD is the first major real-world dataset specifically designed for multi-robot collaborative perception using a heterogeneous team of air and ground robots. Unlike previous datasets focused on mapping (SLAM), CoPeD provides overlapping sensor views and "zero-shot" semantic annotations to help researchers develop algorithms that allow robots to "help" each other see through noise and occlusions.

What Problem Does This Paper Address?

While single-robot perception is highly advanced, collaborative perception—where a network of robots shares data to create a more resilient "collective" view—remains underdeveloped.

The authors identify two main bottlenecks:

  1. The SLAM Bias: Most existing multi-robot datasets are built for Simultaneous Localization and Mapping (SLAM). In SLAM, the goal is to maximize coverage (see as much new ground as possible). In collaborative perception, you actually want overlapping views so that one robot can compensate for another’s sensor noise, occlusions, or failures.
  2. The Simulation Gap: Many collaborative datasets (especially in the autonomous driving/V2V space) are purely simulated. They miss the messy reality of sensor jitter, lighting changes, and communication delays found in the wild.

Methods

The researchers deployed a diverse team: two heavy-duty ground robots (Clearpath Warthog and Jackal) and two agile aerial drones (NYU ARPL Race robots).

The Tech Stack

  • Heterogeneous Sensors: The team utilized 3D LiDAR, IMUs, GNSS (GPS), RGBD cameras, and high-resolution stereo cameras.
  • Cross-Robot Calibration: To ensure the robots knew exactly where they were relative to each other, they used a combination of GPS, Visual-Inertial Odometry (VIO), and AprilTags (visual fiducial markers) mounted on the ground robots for the drones to spot from above.
  • Foundation Model Annotations: Instead of manual labeling, the authors used "Foundation Models" (large-scale AI models like SAM and Grounding DINO) to generate "zero-shot" semantic segmentation masks and depth maps. They then used temporal propagation to ensure these labels stayed consistent across video frames.

Datasets

The core contribution is the CoPeD Dataset, which includes:

  • Indoor Sequences: Captured in a 38m x 60m lab environment (NYUARPL) with tables, chairs, and complex occlusions.
  • Outdoor Sequences: Captured in forest and residential-style environments (HOUSE and FOREST), featuring trees, rocks, and varying terrain.
  • Comparison: The paper compares CoPeD against 11 existing datasets (like Lamp 2.0 and Opv2v), highlighting that CoPeD is unique in offering real-world, air-ground, multi-modal data specifically for perception rather than just SLAM.

Key Insights and Ideas

  • Overlap is a Feature: By ensuring sensor views overlap, the dataset allows for "perception compensation." If a ground robot's camera is obscured by a bush, the drone hovering above can provide the missing data.
  • Air-Ground Synergy: Ground robots carry heavy, high-accuracy sensors (like 64-beam LiDAR), while aerial robots provide a "bird's eye" context that ground units lack. This complementarity is the "secret sauce" for robust perception.
  • Automated Labeling: By using foundation models for annotation, the researchers demonstrate a scalable way to create high-quality training data for robotics without thousands of hours of manual clicking.

Key Results

The authors validated the dataset using Graph Neural Networks (GNNs) for collaborative tasks:

  • Collaborative Depth Estimation: When one robot's sensor was "corrupted" (simulating a failure), the multi-robot system could recover the depth data by communicating feature maps with its teammates.
  • Semantic Robustness: Collaborative perception significantly improved the detection of far-away objects (like distant houses or trees) compared to a single-robot baseline.
  • Real-World Noise: The dataset successfully captures "disturbances in the wild," such as GPS drift in forests and IMU inconsistencies due to temperature changes.

Limitations and Future Work

While CoPeD is a massive step forward, the authors acknowledge a few areas for growth:

  • 2D vs 3D: Currently, the semantic annotations are primarily 2D. The team plans to enrich the dataset with comprehensive 3D semantic labels in the future.
  • Environment Variety: While they cover indoor and forest settings, they aim to expand into even more diverse environmental contexts.
  • Real-to-Sim: One major goal is using this real-world data to improve the fidelity of robotic simulators, creating a "virtuous cycle" between the two.

For researchers in multi-agent systems, CoPeD represents a new "playground" for testing how robots can truly work together to understand the world.

Check out the project: * Video: https://youtu.be/3xGmo0mjVzg * Dataset: https://github.com/arplaboratory/CoPeD

Summary updated

Abstract

In the past decade, although single-robot perception has made significant advancements, the exploration of multi-robot collaborative perception remains largely unexplored. This involves fusing compressed, intermittent, limited, heterogeneous, and asynchronous environmental information across multiple robots to enhance overall perception, despite challenges like sensor noise, occlusions, and sensor failures. One major hurdle has been the lack of real-world datasets. This letter presents a pioneering and comprehensive real-world multi-robot collaborative perception dataset to boost research in this area. Our dataset leverages the untapped potential of air-ground robot collaboration featuring distinct spatial viewpoints, complementary robot mobilities, coverage ranges, and sensor modalities. It features raw sensor inputs, pose estimation, and optional high-level perception annotation, thus accommodating diverse research interests. Compared to existing datasets predominantly designed for Simultaneous Localization and Mapping (SLAM), our setup ensures a diverse range and adequate overlap of sensor views to facilitate the study of multi-robot collaborative perception algorithms. We demonstrate the value of this dataset qualitatively through multiple collaborative perception tasks. We believe this work will unlock the potential research of high-level scene understanding through multi-modal collaborative perception in multi-robot settings.
BibTeX
@article{zhou2024coped,
  author = {Yang Zhou and Long Quang and Carlos Nieto-Granda and Giuseppe Loianno},
  title = {{CoPeD-Advancing Multi-Robot Collaborative Perception: A Comprehensive Dataset in Real-World Environments}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=mHf7SLKQKN},
}
Figure