Collaborative Dynamic 3D Scene Graphs for Open-Vocabulary Urban Scene Understanding

Tim Steinke, Martin Büchner, Niclas Vödisch, Abhinav ValadaarXiv 2025 · arXiv

TL;DR

CURB-OSG is a novel centralized framework that constructs open-vocabulary, dynamic 3D scene graphs of large-scale urban environments by collaborating across multiple robotic agents. Crucially, it operates on real-world data without requiring any prior knowledge of the agents' initial relative positions, bridging the gap between raw geometric mapping and high-level semantic scene understanding.

What Problem Does This Paper Address?

For autonomous vehicles and mobile robots to navigate safely and plan efficiently, they need maps that are both geometrically accurate and semantically rich. Traditional mapping methods like voxel grids or point clouds tell a robot where space is occupied, but not what occupies it or how those objects relate to each other.

3D Scene Graphs solve this by organizing environments into a sparse, hierarchical structure (e.g., decomposing a city into roads, intersections, static landmarks, and dynamic obstacles). However, existing methods suffer from three major bottlenecks:

  1. Closed-Vocabulary Constraints: They rely on object detectors trained on fixed categories (e.g., only detecting "car" or "pedestrian"), failing in open-world deployments where unexpected objects appear.
  2. Assumption of Known Poses: Multi-agent systems usually assume the initial relative positions of all robots are known, or they rely heavily on GNSS/GPS.
  3. Simulation Bias: Most complex frameworks are only validated in pristine simulators rather than messy, real-world urban environments.

CURB-OSG addresses these challenges by combining open-vocabulary foundation models with a collaborative SLAM backend capable of "blind" inter-agent loop closure detection.

Methods

The architecture of CURB-OSG is split between onboard processing on individual agents and a centralized server that aggregates the data.

Overview of the CURB-OSG architecture showing processing on N agents and centralized server-side graph construction.

1. Collaborative SLAM Backend

Onboard agents compute local LiDAR odometry using Fast GICP and send keyframes to the server. Because initial alignments are unknown, the server utilizes Scan Context—a global point cloud descriptor—to blindly detect loop closures between different agents. Once a match is found, a global pose graph optimization (using $g^2o$) aligns all agents into a unified coordinate frame.

2. Open-Vocabulary Perception

Instead of traditional semantic segmentation, the pipeline leverages vision-language foundation models:

  • Static Objects: Images are processed using Grounding DINO prompted with text classes relevant to urban scenes. These bounding boxes are converted into pixel-exact masks using TAP (Tokenize Anything via Prompting) and reprojected onto the 3D LiDAR point clouds.
  • Dynamic Objects: To track moving obstacles across frames, the agents combine Grounding DINO with MASA (Matching Anything by Segmenting Anything) to generate temporally consistent instance IDs before projecting them into 3D.

3. Scene Graph Generation

The server ingests the optimized pose graph and the 3D object observations to build a 5-layer hierarchy:

  • Layer A (Environment): The root node.
  • Layer B (Roads & Intersections): Extracted automatically using a heuristic based on trajectory "disfluency" (sharp turns) clustered via DBSCAN.
  • Layer C (Static Objects): Fused 3D bounding boxes of landmarks like traffic lights and signs.
  • Layer D (Dynamic Objects): Global tracking paths of moving or temporarily parked vehicles/pedestrians.
  • Layer E (Pose Graph & Point Clouds): The raw metric foundation.

Datasets

The authors evaluated their approach using real-world data from the Oxford Radar RobotCar Dataset. They selected three distinct sequences spanning a combined travel distance of 28 km to simulate a multi-agent exploration scenario.

To evaluate the open-vocabulary semantic accuracy, the authors also introduced a new benchmark: 100 human-annotated panoptic images following the Cityscapes category convention, which they have released publicly alongside their codebase.

Key Insights and Ideas

  • The Power of Multi-Agent Consensus: When multiple agents observe the same area, semantic ambiguities disappear. By enforcing a consensus rule (requiring an object to be seen by all agents), the system achieved flawless precision for static object detection.
  • The Dynamic SLAM Paradox: Intuitively, removing moving cars and pedestrians should improve SLAM tracking. However, the authors found that dynamic object removal did not improve mapping quality. This is because the heavy vision-language tracking models (MASA) run at a lower frequency than the LiDAR odometry, meaning the tracking updates couldn't keep pace with the fast scan-matching backend.
  • Heuristic Topology is Fast but Blind: Using trajectory disfluency to find intersections is computationally lightweight and highly robust, but it has a catch: it can only detect an intersection if at least one agent actually makes a turn there. Straight-line passes leave intersections undetected.

Key Results

Mapping Performance

Collaborative mapping steadily reduced the Absolute Trajectory Error (ATE) as more agents were introduced, proving that the blind inter-agent loop closures successfully corrected global drift.

  • 1 Agent: ATE of $17.8 \pm 11.6$ meters
  • 3 Agents: ATE dropped to $15.7 \pm 10.2$ meters
Evolution of the mean absolute trajectory error (ATE) over time, demonstrating sharp drops when inter-agent loop closures are found.

Object Fusion Accuracy

Using Grounding DINO + TAP significantly outperformed alternative open-vocabulary baselines like MaskCLIP:

  • Ours (Grounding DINO + TAP): 31.83% mIoU, 88.22% Precision, 14.92% Recall
  • MaskCLIP Baseline: 27.01% mIoU, 68.29% Precision, 8.54% Recall

When scaling to 3 agents and evaluating objects observed by all agents, the framework achieved 100% Precision (with an mIoU of 38.15%), showing that multi-agent collaboration eliminates false positives.

Limitations and Future Work

While CURB-OSG demonstrates impressive real-world capabilities, the authors note a few key limitations:

  1. Computational Bottleneck: The open-vocabulary tracking pipeline (MASA) is too slow to run at the same high frequency as the LiDAR sensors, creating a latency bottleneck.
  2. Intersection Detection Limits: The topological road graph relies entirely on vehicles turning. If an agent drives straight through a crossroad, the system cannot infer the intersection's existence.

Future Directions: The team aims to optimize the dynamic object tracking pipeline and explore using the semantic entities within the scene graph itself to help trigger loop closures, creating a tighter loop between perception and localization.

Summary updated

Abstract

Mapping and scene representation are fundamental to reliable planning and navigation in mobile robots. While purely geometric maps using voxel grids allow for general navigation, obtaining up-to-date spatial and semantically rich representations that scale to dynamic large-scale environments remains challenging. In this work, we present CURB-OSG, an open-vocabulary dynamic 3D scene graph engine that generates hierarchical decompositions of urban driving scenes via multi-agent collaboration. By fusing the camera and LiDAR observations from multiple perceiving agents with unknown initial poses, our approach generates more accurate maps compared to a single agent while constructing a unified open-vocabulary semantic hierarchy of the scene. Unlike previous methods that rely on ground truth agent poses or are evaluated purely in simulation, CURB-OSG alleviates these constraints. We evaluate the capabilities of CURB-OSG on real-world multi-agent sensor data obtained from multiple sessions of the Oxford Radar RobotCar dataset. We demonstrate improved mapping and object prediction accuracy through multi-agent collaboration as well as evaluate the environment partitioning capabilities of the proposed approach. To foster further research, we release our code and supplementary material at https://ov-curb.cs.uni-freiburg.de.
BibTeX
@misc{steinke2025collaborative,
  author = {Tim Steinke and Martin Büchner and Niclas Vödisch and Abhinav Valada},
  title = {{Collaborative Dynamic 3D Scene Graphs for Open-Vocabulary Urban Scene Understanding}},
  year = {2025},
  eprint = {2503.08474},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.08474},
}
Figure