RNGDet++: Road Network Graph Detection by Transformer With Instance Segmentation and Multi-Scale Features Enhancement

Zhenhua Xu, Zhenhua Xu 0003, Yuxuan Liu, Yuxiang Sun, Ming Liu, Lujia WangIEEE RA-L 2023 · DOI

TL;DR

This paper introduces RNGDet++, an improved method for automatically detecting road network graphs from aerial imagery. By incorporating an instance segmentation head and leveraging multi-scale features from its backbone, RNGDet++ achieves state-of-the-art performance in accurately representing road topology, crucial for autonomous driving navigation.

What Problem Does This Paper Address?

Creating detailed road network maps, essential for autonomous driving (e.g., global route planning, navigation), is currently a manual and labor-intensive process. While automatic methods exist, they often struggle with the complex topology of road networks, such as intersections and overpasses, or rely on cumbersome post-processing steps. Existing state-of-the-art methods like RNGDet, while effective, don't fully exploit the rich, multi-scale information available in the deep features extracted by their convolutional backbones, limiting their potential for further improvement. This paper aims to bridge this gap by enhancing RNGDet to better utilize these multi-scale features and improve supervision during training.

Methods

RNGDet++ builds upon the Transformer-based RNGDet architecture. The core improvements are:

  • Instance Segmentation Head: An additional head is added to predict instance segmentation masks for road segments ahead of the agent's current position. This provides a stronger supervisory signal during training, helping the network better understand road geometry and topology, especially in complex scenarios.
  • Multi-Scale Feature Fusion: Unlike the original RNGDet, which only used the deepest feature layer from its ResNet backbone, RNGDet++ integrates features from all four levels of the backbone. This is achieved through a Feature Pyramid Network (FPN) approach, allowing the model to capture both fine-grained local details and broader contextual information, leading to a more comprehensive understanding of the road network.
  • Iterative Graph Detection: The system still employs an iterative approach, controlled by an agent, to detect the road network graph. This agent moves through the scene, predicting vertices and edges step-by-step, guided by the extracted features and segmentation outputs. A FIFO buffer manages initial candidates and intermediate predictions to handle complex road structures.
  • Imitation Learning: The model is trained using imitation learning, where an "expert" (a BFS graph traversal algorithm) generates ground truth trajectories, and the network learns to mimic this expert behavior.

Datasets

The paper evaluates RNGDet++ on two publicly available datasets:

  • City-scale dataset: This dataset, released by Sat2Graph, contains 180 RGB aerial images (2048x2048 pixels) with ground truth road networks in vector format. It's split into train/valid/test sets (144/9/27 tiles).
  • SpaceNet dataset: This dataset comprises 2551 RGB aerial images (400x400 pixels) with a 1m/pixel resolution, focusing on smaller regions. It's split into train/valid/test sets (2042/127/382 images).

Both datasets provide aerial imagery suitable for road network detection and are used to comprehensively evaluate the proposed method against baselines.

Key Insights and Ideas

  • Multi-scale features are crucial for road topology: The paper strongly demonstrates that simply using the deepest feature layer is insufficient. Integrating features from various scales significantly improves the model's ability to capture complex road structures and global topology, as evidenced by the improved APLS scores.
  • Instance segmentation as a supervisor: Adding an instance segmentation head, even for road segments ahead of the current agent position, acts as a powerful supervisor. It forces the network to learn more robust spatial and topological representations, leading to better overall graph detection.
  • Iterative detection with buffer management is effective for complex graphs: The iterative approach, combined with a FIFO buffer to manage potential road splits and intersections, proves effective in navigating and reconstructing intricate road networks.

Key Results

RNGDet++ achieves state-of-the-art performance on both the city-scale and SpaceNet datasets, outperforming previous methods in terms of both TOPO (local subgraph similarity) and APLS (average path length similarity) metrics.

  • On the city-scale dataset, RNGDet++ achieved a TOPO-F1 score of 78.44% and an APLS score of 67.76%.
  • On the SpaceNet dataset, it achieved a TOPO-F1 score of 82.51% and an APLS score of 67.73%.

Qualitative visualizations show that RNGDet++ produces road network graphs with more accurate structures and correctness compared to previous methods like Sat2Graph and RNGDet. Ablation studies confirm that both the instance segmentation head and the multi-scale feature fusion contribute significantly to the performance gains.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Lower Efficiency: RNGDet++ has a longer inference time compared to RNGDet due to the processing of multi-scale features and the additional segmentation head. However, they argue this is acceptable for offline tasks like road network mapping. Future work aims to simplify the network and use lighter structures.
  • Complex Intersections and Overpasses: While improved, the method still struggles with extremely complex road intersections and overpasses. Imitation learning can also be sensitive to incorrect predictions in these scenarios. Future work will focus on optimizing the training strategy and exploring more powerful backbones to enhance reasoning capabilities.

Summary updated

BibTeX
@article{xu2023rngdet,
  author = {Zhenhua Xu and Zhenhua Xu and Yuxuan Liu and Yuxiang Sun and Ming Liu and Lujia Wang},
  title = {{RNGDet++: Road Network Graph Detection by Transformer With Instance Segmentation and Multi-Scale Features Enhancement}},
  journal = {IEEE RA-L},
  year = {2023},
  doi = {10.1109/lra.2023.3264723},
}
Figure