MR-COGraphs: Communication-Efficient Multi-Robot Open-Vocabulary Mapping System via 3D Scene Graphs

Qiuyi Gu, Zhaocheng Ye, Jincheng Yu, Jiahao Tang, Tinghao Yi, Yuhan Dong, Jian Wang, Jinqiang Cui, Xinlei Chen, Yu Wang

TL;DR

MR-COGraphs introduces a communication-efficient framework for multi-robot open-vocabulary mapping by representing environments as 3D Scene Graphs (3DSGs). By employing a data-driven encoder to compress high-dimensional semantic features by over 95%, the system reduces total map data volume by up to 89% while maintaining the ability to perform complex, language-based object queries.

What Problem Does This Paper Address?

In the world of robotics, "open-vocabulary" mapping is the gold standard. Unlike traditional maps that only recognize a fixed set of labels (e.g., "chair," "table"), open-vocabulary systems leverage foundation models like CLIP to understand virtually any text query (e.g., "the blue velvet sofa").

However, there is a massive catch: data volume. To support these queries, robots typically store high-dimensional feature vectors (often 512 dimensions or more) for every point or object in the scene. In a multi-robot setup, where machines must constantly share their maps to coordinate, this creates a massive communication bottleneck. Sharing a semantic map of even a small room can require gigabytes of data, which is impractical for real-time operation over standard wireless networks.

Methods

The authors propose MR-COGraphs, a system built on three main pillars:

  1. COGraph Representation: Instead of dense point clouds, the environment is modeled as a 3D Scene Graph. Nodes represent individual objects (containing semantic features, positions, and bounding boxes), and edges represent spatial relationships.
  2. Data-driven Feature Compression: This is the "secret sauce." The system uses a lightweight Multi-Layer Perceptron (MLP) encoder to compress 512-dimensional CLIP features down to just 3 dimensions before transmission. On the receiving end, a corresponding decoder reconstructs the original 512-dim vector. This allows robots to "whisper" semantic information that previously required a "shout."
  3. Semantic-based Merging: To combine maps from different robots, the system uses these reconstructed features for place recognition. By comparing the semantic "fingerprints" of objects, robots can determine if they are looking at the same area and calculate the necessary spatial transformation to merge their local maps into a global one.

Datasets

The researchers validated their framework using a mix of simulated and physical environments:

  • Replica Dataset: A standard benchmark for high-quality 3D indoor reconstructions (specifically the room0, office2, and apartment2 scenes).
  • Isaac Sim (NVIDIA): Used to create custom multi-room environments (Isaac Small and Isaac Large) to test collaborative mapping and navigation.
  • Real-world Environment: A 9m x 9m three-room office space mapped using two robots equipped with iPhones for sensing.
  • ImageNet: Used to pre-train the feature encoder/decoder modules.

Key Insights and Ideas

  • The Power of 3 Dimensions: Perhaps the most surprising finding is that semantic information can be squashed from 512 dimensions to 3 and still be reconstructed accurately enough for object retrieval. While some nuance is lost (the authors show a "panda" toy being misidentified as a "pillow" after compression), the system remains highly functional for most household objects.
  • General vs. Domain Encoding: The authors found that an encoder trained on a broad variety of images (general-encode) actually performs better for open-vocabulary tasks than one trained on a narrow set of environment-specific images (domain-encode). Generalization is key for "open" understanding.
  • Communication vs. Computation Trade-off: By moving the heavy lifting to a pre-trained encoder/decoder, the system drastically reduces the "cost" of talking between robots, making multi-robot semantic mapping viable in bandwidth-constrained environments.

Key Results

  • Massive Data Reduction: Compared to existing baselines like ConceptGraphs, MR-COGraphs reduced the total data volume by 80.11% to 89.80%.
  • Feature Efficiency: The specific feature compression strategy reduced the communication overhead of the semantic vectors themselves by 95.72%.
  • Real-time Performance: On a standard PC, the segmentation and mapping pipeline runs in roughly 1.5 seconds per frame, while the compression/decompression happens in a fraction of a millisecond (0.174ms).
  • Query Accuracy: Despite the heavy compression, the system maintained a high "Object Finding Rate" ($R_{obj}$) and competitive query success rates ($R@k$), proving that the compressed "sketch" of the semantics is sufficient for robot tasks.

Limitations and Future Work

While impressive, the system has a few hurdles to clear:

  • Reconstruction Artifacts: As noted with the "panda" example, extreme compression can lead to semantic blurring between objects with similar visual features.
  • Rotation Dependence: The current merging strategy assumes robots have access to a common orientation (like a compass). Future iterations might need to handle arbitrary rotations without external sensors.
  • Static Scenes: The current framework focuses on static environments. Integrating dynamic object tracking into a 3D Scene Graph remains a complex challenge for future research.

By slashing the data requirements of semantic mapping, MR-COGraphs brings us one step closer to teams of robots that can truly understand—and discuss—the world around them.

Summary updated

BibTeX
@article{gu2025mr,
  author = {Qiuyi Gu and Zhaocheng Ye and Jincheng Yu and Jiahao Tang and Tinghao Yi and Yuhan Dong and Jian Wang and Jinqiang Cui and Xinlei Chen and Yu Wang},
  title = {{MR-COGraphs: Communication-Efficient Multi-Robot Open-Vocabulary Mapping System via 3D Scene Graphs}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
  doi = {10.1109/lra.2025.3561569},
}
Figure