TL;DR
This paper introduces CoopSR, the first benchmark for multi-robot cooperative egocentric spatial reasoning, alongside EgoTeam, a massive dataset of over 114k QA pairs. To tackle this task, the authors propose SP-CoR, a novel framework that uses spectral frame sampling and physics-informed prompt distillation to effectively fuse multi-view video streams. Remarkably, SP-CoR learns from privileged robot-pose data during training but requires only raw egocentric videos at test time, outperforming 22 standard MLLM baselines.
What Problem Does This Paper Address?
While Multimodal Large Language Models (MLLMs) excel at understanding video from a single perspective, real-world embodied AI increasingly relies on teams of robots working together (e.g., search-and-rescue squads or warehouse fleets). In these scenarios, no single robot has a complete view of the environment; each agent sees only a partial, noisy, egocentric slice of the scene.
Answering a simple team-level question like "Which robot is pushing the object?" or "Can any robot see the blocked doorway?" requires cross-view grounding, spatial relation reasoning, and resolving contradictory observations. Simply concatenating video streams from $N$ robots doesn't scale—it floods the model with redundant visual tokens and fails to teach the model how to collaboratively reason across different viewpoints.
Methods
The authors introduce SP-CoR (Spectral and Physics-Informed Cooperative Reasoner), an MLLM architecture specifically designed to handle multi-robot view fusion through three main components:
- Spectral Energy-Aware Frame Sampler (SE-MR2FS): To avoid token explosion, this training-free, two-stage sampler first filters frames based on query relevance using CLIP. It then refines the selection by applying a Fast Fourier Transform (FFT) over the visual features to rank frames by temporal frequency energy, ensuring that dynamic, action-heavy moments are preserved.
- Spectral and Physics-Informed Multi-Robot Fusion (SPI-MRF): Instead of naively stacking tokens, this module explicitly models cross-view dependencies. It combines visual states with pose, spectral, and robot-role embeddings using cross-attention, aggregating them into a unified "global team belief" via reliability-aware pooling.
- Physics-Aligned Prompt-Space Distillation (PAPSD): This is the framework's secret sauce. During training in a simulator, a "teacher" model uses privileged ground-truth robot poses to create physics-based prompts. The student model is trained to distill these physical and geometric priors directly into its prefix prompt space. At test time, the teacher is discarded, allowing the model to perform physics-aware spatial reasoning using only egocentric videos without needing external localization.
Datasets
The authors established a comprehensive evaluation suite:
- EgoTeam Dataset: Contains 114,227 QA pairs across 19 question types, categorized into four progressive reasoning tiers (from local egocentric QA to team-level counterfactual coordination).
- Simulated Environments: Generated using Habitat and iGibson, featuring variable team sizes ($N$ = 2, 3, or 4 robots).
- Real-World Test Set: Consists of 2,326 QA pairs collected in a physical lab using two Unitree quadruped robots tracked by motion-capture systems to validate sim-to-real generalization.
Key Insights and Ideas
- Prompt-Space vs. Output-Space Distillation: Instead of forcing the student model to mimic the teacher's final text answers, PAPSD distills geometric knowledge directly into the latent prompt space. This approach proves highly sample-efficient because it forces the student to recover the underlying physical structure of the scene.
- FFT on Embeddings Filters Noise: Computing the FFT on high-level CLIP embeddings rather than raw pixels allows the frame sampler to capture semantically meaningful motion (like a robot moving or an object being pushed) while remaining completely invariant to pixel-level noise or lighting changes.
- Robustness to Viewpoint Redundancy: Standard MLLMs degrade sharply when team sizes scale up due to visual redundancy and conflicting perspectives. SP-CoR's reliability-aware pooling explicitly weighs the variance of each robot's visual evidence, successfully mitigating this bottleneck.
Key Results
SP-CoR was evaluated against 22 MLLM baselines (including zero-shot, fine-tuned, and retrieval-augmented variants) and demonstrated superior performance:
- State-of-the-Art Accuracy: SP-CoR (built on a Qwen2.5-VL backbone) achieved an average accuracy of 70.55% on Habitat and 70.82% on iGibson, outperforming the strongest fine-tuned baseline by +3.87% and +7.12%, respectively.
- Cross-Team-Size Generalization: When trained on smaller teams ($N$=2, 3) and tested on larger, unseen teams ($N$=4), SP-CoR retained its edge, outperforming the best baseline by 14.42% on iGibson.
- Sim-to-Real Success: In physical robot trials, SP-CoR generalized remarkably well to real-world viewpoint changes and partial occlusions, outperforming commercial models like GPT-4o-mini and standard SFT baselines.
Limitations and Future Work
While CoopSR pushes the boundaries of multi-agent perception, the authors note that the real-world test set is still relatively constrained in terms of scene diversity, robot types, and team sizes ($N$=2 quadrupeds). Additionally, the current framework relies on synchronized video streams, which might not always be guaranteed in real-world deployments with network latency.
Future work will focus on scaling the benchmark to larger physical robot fleets, exploring more open-ended, safety-critical coordination tasks, and handling asynchronous or communication-constrained multi-robot streams.
Summary updated