TL;DR
CURB-OSG is a novel centralized framework that constructs open-vocabulary, dynamic 3D scene graphs of large-scale urban environments by collaborating across multiple robotic agents. Crucially, it operates on real-world data without requiring any prior knowledge of the agents' initial relative positions, bridging the gap between raw geometric mapping and high-level semantic scene understanding.
What Problem Does This Paper Address?
For autonomous vehicles and mobile robots to navigate safely and plan efficiently, they need maps that are both geometrically accurate and semantically rich. Traditional mapping methods like voxel grids or point clouds tell a robot where space is occupied, but not what occupies it or how those objects relate to each other.
3D Scene Graphs solve this by organizing environments into a sparse, hierarchical structure (e.g., decomposing a city into roads, intersections, static landmarks, and dynamic obstacles). However, existing methods suffer from three major bottlenecks:
- Closed-Vocabulary Constraints: They rely on object detectors trained on fixed categories (e.g., only detecting "car" or "pedestrian"), failing in open-world deployments where unexpected objects appear.
- Assumption of Known Poses: Multi-agent systems usually assume the initial relative positions of all robots are known, or they rely heavily on GNSS/GPS.
- Simulation Bias: Most complex frameworks are only validated in pristine simulators rather than messy, real-world urban environments.
CURB-OSG addresses these challenges by combining open-vocabulary foundation models with a collaborative SLAM backend capable of "blind" inter-agent loop closure detection.
Methods
The architecture of CURB-OSG is split between onboard processing on individual agents and a centralized server that aggregates the data.
1. Collaborative SLAM Backend
Onboard agents compute local LiDAR odometry using Fast GICP and send keyframes to the server. Because initial alignments are unknown, the server utilizes Scan Context—a global point cloud descriptor—to blindly detect loop closures between different agents. Once a match is found, a global pose graph optimization (using $g^2o$) aligns all agents into a unified coordinate frame.
2. Open-Vocabulary Perception
Instead of traditional semantic segmentation, the pipeline leverages vision-language foundation models:
- Static Objects: Images are processed using Grounding DINO prompted with text classes relevant to urban scenes. These bounding boxes are converted into pixel-exact masks using TAP (Tokenize Anything via Prompting) and reprojected onto the 3D LiDAR point clouds.
- Dynamic Objects: To track moving obstacles across frames, the agents combine Grounding DINO with MASA (Matching Anything by Segmenting Anything) to generate temporally consistent instance IDs before projecting them into 3D.
3. Scene Graph Generation
The server ingests the optimized pose graph and the 3D object observations to build a 5-layer hierarchy:
- Layer A (Environment): The root node.
- Layer B (Roads & Intersections): Extracted automatically using a heuristic based on trajectory "disfluency" (sharp turns) clustered via DBSCAN.
- Layer C (Static Objects): Fused 3D bounding boxes of landmarks like traffic lights and signs.
- Layer D (Dynamic Objects): Global tracking paths of moving or temporarily parked vehicles/pedestrians.
- Layer E (Pose Graph & Point Clouds): The raw metric foundation.
Datasets
The authors evaluated their approach using real-world data from the Oxford Radar RobotCar Dataset. They selected three distinct sequences spanning a combined travel distance of 28 km to simulate a multi-agent exploration scenario.
To evaluate the open-vocabulary semantic accuracy, the authors also introduced a new benchmark: 100 human-annotated panoptic images following the Cityscapes category convention, which they have released publicly alongside their codebase.
Key Insights and Ideas
- The Power of Multi-Agent Consensus: When multiple agents observe the same area, semantic ambiguities disappear. By enforcing a consensus rule (requiring an object to be seen by all agents), the system achieved flawless precision for static object detection.
- The Dynamic SLAM Paradox: Intuitively, removing moving cars and pedestrians should improve SLAM tracking. However, the authors found that dynamic object removal did not improve mapping quality. This is because the heavy vision-language tracking models (MASA) run at a lower frequency than the LiDAR odometry, meaning the tracking updates couldn't keep pace with the fast scan-matching backend.
- Heuristic Topology is Fast but Blind: Using trajectory disfluency to find intersections is computationally lightweight and highly robust, but it has a catch: it can only detect an intersection if at least one agent actually makes a turn there. Straight-line passes leave intersections undetected.
Key Results
Mapping Performance
Collaborative mapping steadily reduced the Absolute Trajectory Error (ATE) as more agents were introduced, proving that the blind inter-agent loop closures successfully corrected global drift.
- 1 Agent: ATE of $17.8 \pm 11.6$ meters
- 3 Agents: ATE dropped to $15.7 \pm 10.2$ meters
Object Fusion Accuracy
Using Grounding DINO + TAP significantly outperformed alternative open-vocabulary baselines like MaskCLIP:
- Ours (Grounding DINO + TAP): 31.83% mIoU, 88.22% Precision, 14.92% Recall
- MaskCLIP Baseline: 27.01% mIoU, 68.29% Precision, 8.54% Recall
When scaling to 3 agents and evaluating objects observed by all agents, the framework achieved 100% Precision (with an mIoU of 38.15%), showing that multi-agent collaboration eliminates false positives.
Limitations and Future Work
While CURB-OSG demonstrates impressive real-world capabilities, the authors note a few key limitations:
- Computational Bottleneck: The open-vocabulary tracking pipeline (MASA) is too slow to run at the same high frequency as the LiDAR sensors, creating a latency bottleneck.
- Intersection Detection Limits: The topological road graph relies entirely on vehicles turning. If an agent drives straight through a crossroad, the system cannot infer the intersection's existence.
Future Directions: The team aims to optimize the dynamic object tracking pipeline and explore using the semantic entities within the scene graph itself to help trigger loop closures, creating a tighter loop between perception and localization.
Summary updated