Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models

Kunyu Peng, Zhikun Zhou, Kailun Yang, Di Wen, Ruiping Liu, Yufan Chen, Junwei Zheng, Hao Shi, Yi Zhou, M. Saquib Sarfraz, Danda Pani Paudel, Luc Van GoolarXiv 2026 · arXiv

TL;DR

This paper introduces CoopSR, the first benchmark for multi-robot cooperative egocentric spatial reasoning, alongside EgoTeam, a massive dataset of over 114k QA pairs. To tackle this task, the authors propose SP-CoR, a novel framework that uses spectral frame sampling and physics-informed prompt distillation to effectively fuse multi-view video streams. Remarkably, SP-CoR learns from privileged robot-pose data during training but requires only raw egocentric videos at test time, outperforming 22 standard MLLM baselines.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) excel at understanding video from a single perspective, real-world embodied AI increasingly relies on teams of robots working together (e.g., search-and-rescue squads or warehouse fleets). In these scenarios, no single robot has a complete view of the environment; each agent sees only a partial, noisy, egocentric slice of the scene.

Answering a simple team-level question like "Which robot is pushing the object?" or "Can any robot see the blocked doorway?" requires cross-view grounding, spatial relation reasoning, and resolving contradictory observations. Simply concatenating video streams from $N$ robots doesn't scale—it floods the model with redundant visual tokens and fails to teach the model how to collaboratively reason across different viewpoints.

Methods

The authors introduce SP-CoR (Spectral and Physics-Informed Cooperative Reasoner), an MLLM architecture specifically designed to handle multi-robot view fusion through three main components:

  1. Spectral Energy-Aware Frame Sampler (SE-MR2FS): To avoid token explosion, this training-free, two-stage sampler first filters frames based on query relevance using CLIP. It then refines the selection by applying a Fast Fourier Transform (FFT) over the visual features to rank frames by temporal frequency energy, ensuring that dynamic, action-heavy moments are preserved.
  2. Spectral and Physics-Informed Multi-Robot Fusion (SPI-MRF): Instead of naively stacking tokens, this module explicitly models cross-view dependencies. It combines visual states with pose, spectral, and robot-role embeddings using cross-attention, aggregating them into a unified "global team belief" via reliability-aware pooling.
  3. Physics-Aligned Prompt-Space Distillation (PAPSD): This is the framework's secret sauce. During training in a simulator, a "teacher" model uses privileged ground-truth robot poses to create physics-based prompts. The student model is trained to distill these physical and geometric priors directly into its prefix prompt space. At test time, the teacher is discarded, allowing the model to perform physics-aware spatial reasoning using only egocentric videos without needing external localization.
Figure 2: Overview of SP-CoR for CoopSR: a query-guided spectral energy sampler selects in- formative multi-robot egocentric frames, while spectral- and physics-informed fusion with prompt distillation integrates robot-view evidence for cooperative spatial reasoning.

Datasets

The authors established a comprehensive evaluation suite:

  • EgoTeam Dataset: Contains 114,227 QA pairs across 19 question types, categorized into four progressive reasoning tiers (from local egocentric QA to team-level counterfactual coordination).
  • Simulated Environments: Generated using Habitat and iGibson, featuring variable team sizes ($N$ = 2, 3, or 4 robots).
  • Real-World Test Set: Consists of 2,326 QA pairs collected in a physical lab using two Unitree quadruped robots tracked by motion-capture systems to validate sim-to-real generalization.
Table 1: Comparison of our EgoTeam with other egocentric QA datasets.

Key Insights and Ideas

  • Prompt-Space vs. Output-Space Distillation: Instead of forcing the student model to mimic the teacher's final text answers, PAPSD distills geometric knowledge directly into the latent prompt space. This approach proves highly sample-efficient because it forces the student to recover the underlying physical structure of the scene.
  • FFT on Embeddings Filters Noise: Computing the FFT on high-level CLIP embeddings rather than raw pixels allows the frame sampler to capture semantically meaningful motion (like a robot moving or an object being pushed) while remaining completely invariant to pixel-level noise or lighting changes.
  • Robustness to Viewpoint Redundancy: Standard MLLMs degrade sharply when team sizes scale up due to visual redundancy and conflicting perspectives. SP-CoR's reliability-aware pooling explicitly weighs the variance of each robot's visual evidence, successfully mitigating this bottleneck.

Key Results

SP-CoR was evaluated against 22 MLLM baselines (including zero-shot, fine-tuned, and retrieval-augmented variants) and demonstrated superior performance:

  • State-of-the-Art Accuracy: SP-CoR (built on a Qwen2.5-VL backbone) achieved an average accuracy of 70.55% on Habitat and 70.82% on iGibson, outperforming the strongest fine-tuned baseline by +3.87% and +7.12%, respectively.
  • Cross-Team-Size Generalization: When trained on smaller teams ($N$=2, 3) and tested on larger, unseen teams ($N$=4), SP-CoR retained its edge, outperforming the best baseline by 14.42% on iGibson.
  • Sim-to-Real Success: In physical robot trials, SP-CoR generalized remarkably well to real-world viewpoint changes and partial occlusions, outperforming commercial models like GPT-4o-mini and standard SFT baselines.
Qualitative examples from the real-world test set demonstrating SP-CoR's superior team belief updates and mutual visibility reasoning.

Limitations and Future Work

While CoopSR pushes the boundaries of multi-agent perception, the authors note that the real-world test set is still relatively constrained in terms of scene diversity, robot types, and team sizes ($N$=2 quadrupeds). Additionally, the current framework relies on synchronized video streams, which might not always be guaranteed in real-world deployments with network latency.

Future work will focus on scaling the benchmark to larger physical robot fleets, exploring more open-ended, safety-critical coordination tasks, and handling asynchronous or communication-constrained multi-robot streams.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have made substantial progress in egocentric video understanding, but their ability to reason cooperatively from multiple embodied viewpoints remains largely unexplored. We study this problem through multi-robot cooperative dynamic spatial reasoning, where a model must answer spatial, temporal, visibility, and coordination questions by integrating synchronized egocentric videos from a team of moving robots. To support this setting, we introduce CoopSR, the first benchmark for this task, together with EgoTeam, a multi-robot egocentric QA dataset. EgoTeam contains 114,227 QA pairs spanning 19 question types, four difficulty tiers, and three team sizes in Habitat and iGibson, along with a real-world test set of around 2,326 QAs collected using two quadruped robots. We further propose SP-CoR (Spectral and Physics-Informed Cooperative Reasoner), an MLLM framework for fine-grained cooperative spatial reasoning. SP-CoR combines dynamics-aware multi-robot frame sampling, spectral- and physics-guided view fusion, and physics-aligned prompt distillation, enabling the model to benefit from privileged robot-pose supervision during training while requiring only egocentric videos at test time. Across 22 MLLM baselines, SP-CoR consistently improves cooperative reasoning, outperforming the strongest fine-tuned baseline by +3.87% on Habitat and +7.12% on iGibson. It also shows stronger generalization to unseen team sizes and real-world robot tests. Code can be found at https://github.com/KPeng9510/seeing-together.git.
BibTeX
@misc{peng2026seeing,
  author = {Kunyu Peng and Zhikun Zhou and Kailun Yang and Di Wen and Ruiping Liu and Yufan Chen and Junwei Zheng and Hao Shi and Yi Zhou and M. Saquib Sarfraz and Danda Pani Paudel and Luc Van Gool},
  title = {{Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models}},
  year = {2026},
  eprint = {2605.18431},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.18431},
}

Loading related content...

Figure