Collaborative Dynamic 3D Scene Graphs for Automated Driving

Elias Greve, Martin Büchner, Niclas Vödisch, Wolfram Burgard, Abhinav ValadaarXiv 2023 · DOI · arXiv

TL;DR

CURB-SG (Collaborative URBan Scene Graphs) is a novel framework that allows multiple autonomous vehicles to cooperatively build a hierarchical, queryable 3D semantic map of large-scale urban environments. By leveraging panoptic LiDAR data and a centralized collaborative SLAM backend, it abstracts complex cityscapes into structured layers—ranging from raw point clouds up to roads, intersections, and landmarks—enabling efficient higher-order reasoning for automated driving.

What Problem Does This Paper Address?

High-definition (HD) maps are essential for autonomous driving tasks like localization, perception, and planning. However, traditional HD maps are typically built top-down, requiring arduous manual labeling, or bottom-up using dense voxel grids that are memory-intensive and difficult to query efficiently.

While 3D scene graphs have emerged as a powerful solution to compress and structure spatial data, their application has been almost exclusively confined to indoor environments (structuring scenes into buildings, rooms, and furniture). Translating this concept to the outdoor, dynamic, and large-scale nature of urban driving introduces major challenges: How do we define the "rooms" of a city? How do we handle multi-agent data fusion without the pose graph exploding in size over time?

Methods

The authors introduce CURB-SG, a centralized multi-agent framework structured into several key components:

  1. Collaborative LiDAR SLAM Backend: Multiple agents equipped with LiDAR sensors perform local odometry and panoptic segmentation. They transmit compact keyframe packages (static "stuff" like buildings and roads, and dynamic "thing" instances like vehicles) to a central server. The server detects intra- and inter-agent loop closures to optimize a global pose graph.
  2. Edge Contraction & Node Merging: To ensure long-term scalability and prevent the pose graph from growing indefinitely, the server employs a heuristic-driven edge contraction mechanism. Redundant nodes along the same routes are merged, maintaining a lean and efficient graph structure.
  3. Dynamic Lane Graph Construction: Instead of relying solely on the ego-agents' trajectories, the server extracts the centroids of all observed surrounding vehicles. By filtering and aggregating these trajectories, it dynamically constructs a directed lane graph.
  4. Environment Partitioning: Using the lane graph, the system automatically segregates the cityscape into intersecting and non-intersecting road areas (the outdoor equivalents of indoor rooms and corridors).

The final output is a multi-layered 3D urban scene graph structured into hierarchical layers, as illustrated below:

Overview of the CURB-SG framework, showing data flow from multiple agents to the central server, resulting in a multi-layered 3D Urban Scene Graph.

Datasets

Due to the lack of real-world, multi-agent urban datasets providing panoptic LiDAR scans, the framework was validated using the photorealistic CARLA simulator. The authors evaluated their approach across four distinct environments:

  • Town01 & Town02: Classical two-way roads.
  • Town07: Includes roundabouts.
  • Town10: Features complex, bidirectional multi-lane roads.

Experiments were conducted using configurations of one, two, and three agents alongside 30 to 40 additional simulated non-agent vehicles to mimic realistic traffic behavior.

Key Insights and Ideas

  • The Outdoor "Room" Analogy: The core conceptual breakthrough is using a dynamically generated lane graph to partition open outdoor environments into discrete, manageable spatial zones (intersections and roads), making indoor scene graph concepts applicable to autonomous driving.
  • Crowdsourcing the Map via Traffic Observations: A single autonomous vehicle can only map the lane it drives on. By tracking the panoptic instance IDs of other vehicles on the road, CURB-SG drastically accelerates lane graph completeness, mapping oncoming lanes and cross-streets without ever driving on them.
  • Graph Pruning is Non-Negotiable: Multi-agent mapping generates massive amounts of redundant data. The authors demonstrate that without an explicit edge contraction mechanism, frequent global graph optimization quickly becomes computationally infeasible.

Key Results

  • Faster Exploration & Better Accuracy: Multi-agent collaboration dramatically reduces map exploration time. In town02, three agents achieved 85% map exploration in roughly 600 simulation steps, compared to over 2,000 steps for a single agent. Furthermore, global optimization via inter-agent loop closures consistently reduced localization errors (RMSE).
  • Graph Scalability: The edge contraction mechanism successfully bounded the pose graph size, keeping the number of optimizable graph nodes stable even as the total number of received keyframes continued to climb.
  • Superior Partitioning: The lane-graph-based partitioning method (LG) achieved at least 20% greater precision in intersection detection compared to a morphological image-based skeletonization baseline (SK), proving much more robust to occlusions and sparse LiDAR data.
Visualization of the constructed lane graph in town02, comparing a single-agent scenario to a collaborative three-agent scenario.

Limitations and Future Work

While CURB-SG marks a significant step forward, some limitations remain:

  • Simulation Reliance: The current pipeline relies on ground-truth panoptic labels and known initial agent poses provided by the CARLA simulator. Testing on real-world data with noisy, learned panoptic segmentation models is a critical next step.
  • Centralization Bottleneck: The centralized server architecture introduces bandwidth and single-point-of-failure challenges. The authors note that a decentralized variant operating under real-time constraints is a promising direction for future work.
  • Expanding the Taxonomy: Future iterations aim to incorporate pedestrian information and explicit topological elements like road boundaries to make the scene graph even more actionable for downstream planning tasks.

Summary updated

Abstract

Maps have played an indispensable role in enabling safe and automated driving. Although there have been many advances on different fronts ranging from SLAM to semantics, building an actionable hierarchical semantic representation of urban dynamic scenes and processing information from multiple agents are still challenging problems. In this work, we present Collaborative URBan Scene Graphs (CURB-SG) that enable higher-order reasoning and efficient querying for many functions of automated driving. CURB-SG leverages panoptic LiDAR data from multiple agents to build large-scale maps using an effective graph-based collaborative SLAM approach that detects inter-agent loop closures. To semantically decompose the obtained 3D map, we build a lane graph from the paths of ego agents and their panoptic observations of other vehicles. Based on the connectivity of the lane graph, we segregate the environment into intersecting and non-intersecting road areas. Subsequently, we construct a multi-layered scene graph that includes lane information, the position of static landmarks and their assignment to certain map sections, other vehicles observed by the ego agents, and the pose graph from SLAM including 3D panoptic point clouds. We extensively evaluate CURB-SG in urban scenarios using a photorealistic simulator. We release our code at http://curb.cs.uni-freiburg.de.
BibTeX
@misc{greve2023collaborative,
  author = {Elias Greve and Martin Büchner and Niclas Vödisch and Wolfram Burgard and Abhinav Valada},
  title = {{Collaborative Dynamic 3D Scene Graphs for Automated Driving}},
  year = {2023},
  eprint = {2309.06635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.06635},
}
Figure