Robotic perception has come a long way in the last decade, but most of that progress has happened in a vacuum—specifically, the vacuum of a single robot’s perspective. When you move to a team of robots, the game changes. You have to fuse data that is often compressed, intermittent, heterogeneous (different types of sensors), and asynchronous.
A new paper from researchers at NYU and the U.S. Army Research Laboratory introduces CoPeD (Collaborative Perception Dataset), a comprehensive real-world dataset designed to move multi-robot systems past these hurdles.
TL;DR
CoPeD is the first major real-world dataset specifically designed for multi-robot collaborative perception using a heterogeneous team of air and ground robots. Unlike previous datasets focused on mapping (SLAM), CoPeD provides overlapping sensor views and "zero-shot" semantic annotations to help researchers develop algorithms that allow robots to "help" each other see through noise and occlusions.
What Problem Does This Paper Address?
While single-robot perception is highly advanced, collaborative perception—where a network of robots shares data to create a more resilient "collective" view—remains underdeveloped.
The authors identify two main bottlenecks:
- The SLAM Bias: Most existing multi-robot datasets are built for Simultaneous Localization and Mapping (SLAM). In SLAM, the goal is to maximize coverage (see as much new ground as possible). In collaborative perception, you actually want overlapping views so that one robot can compensate for another’s sensor noise, occlusions, or failures.
- The Simulation Gap: Many collaborative datasets (especially in the autonomous driving/V2V space) are purely simulated. They miss the messy reality of sensor jitter, lighting changes, and communication delays found in the wild.
Methods
The researchers deployed a diverse team: two heavy-duty ground robots (Clearpath Warthog and Jackal) and two agile aerial drones (NYU ARPL Race robots).
The Tech Stack
- Heterogeneous Sensors: The team utilized 3D LiDAR, IMUs, GNSS (GPS), RGBD cameras, and high-resolution stereo cameras.
- Cross-Robot Calibration: To ensure the robots knew exactly where they were relative to each other, they used a combination of GPS, Visual-Inertial Odometry (VIO), and AprilTags (visual fiducial markers) mounted on the ground robots for the drones to spot from above.
- Foundation Model Annotations: Instead of manual labeling, the authors used "Foundation Models" (large-scale AI models like SAM and Grounding DINO) to generate "zero-shot" semantic segmentation masks and depth maps. They then used temporal propagation to ensure these labels stayed consistent across video frames.
Datasets
The core contribution is the CoPeD Dataset, which includes:
- Indoor Sequences: Captured in a 38m x 60m lab environment (NYUARPL) with tables, chairs, and complex occlusions.
- Outdoor Sequences: Captured in forest and residential-style environments (HOUSE and FOREST), featuring trees, rocks, and varying terrain.
- Comparison: The paper compares CoPeD against 11 existing datasets (like Lamp 2.0 and Opv2v), highlighting that CoPeD is unique in offering real-world, air-ground, multi-modal data specifically for perception rather than just SLAM.
Key Insights and Ideas
- Overlap is a Feature: By ensuring sensor views overlap, the dataset allows for "perception compensation." If a ground robot's camera is obscured by a bush, the drone hovering above can provide the missing data.
- Air-Ground Synergy: Ground robots carry heavy, high-accuracy sensors (like 64-beam LiDAR), while aerial robots provide a "bird's eye" context that ground units lack. This complementarity is the "secret sauce" for robust perception.
- Automated Labeling: By using foundation models for annotation, the researchers demonstrate a scalable way to create high-quality training data for robotics without thousands of hours of manual clicking.
Key Results
The authors validated the dataset using Graph Neural Networks (GNNs) for collaborative tasks:
- Collaborative Depth Estimation: When one robot's sensor was "corrupted" (simulating a failure), the multi-robot system could recover the depth data by communicating feature maps with its teammates.
- Semantic Robustness: Collaborative perception significantly improved the detection of far-away objects (like distant houses or trees) compared to a single-robot baseline.
- Real-World Noise: The dataset successfully captures "disturbances in the wild," such as GPS drift in forests and IMU inconsistencies due to temperature changes.
Limitations and Future Work
While CoPeD is a massive step forward, the authors acknowledge a few areas for growth:
- 2D vs 3D: Currently, the semantic annotations are primarily 2D. The team plans to enrich the dataset with comprehensive 3D semantic labels in the future.
- Environment Variety: While they cover indoor and forest settings, they aim to expand into even more diverse environmental contexts.
- Real-to-Sim: One major goal is using this real-world data to improve the fidelity of robotic simulators, creating a "virtuous cycle" between the two.
For researchers in multi-agent systems, CoPeD represents a new "playground" for testing how robots can truly work together to understand the world.
Check out the project: * Video: https://youtu.be/3xGmo0mjVzg * Dataset: https://github.com/arplaboratory/CoPeD
Summary updated