AI Powered Road Network Prediction with Multi-Modal Data

Necip Enes Gengeç, Ergin Tari, Ulas Bagci

Abstract

This study presents an innovative approach for automatic road detection with deep learning, by employing fusion strategies for utilizing both lower-resolution satellite imagery and GPS trajectory data, a concept never explored before. We rigorously investigate both early and late fusion strategies, and assess deep learning based road detection performance using different fusion settings. Our extensive ablation studies assess the efficacy of our framework under diverse model architectures, loss functions, and geographic domains (Istanbul and Montreal). For an unbiased and complete evaluation of road detection results, we use both region-based and boundary-based evaluation metrics for road segmentation. The outcomes reveal that the ResUnet model outperforms U-Net and D-Linknet in road extraction tasks, achieving superior results over the benchmark study using low-resolution Sentinel-2 data. This research not only contributes to the field of automatic road detection but also offers novel insights into the utilization of data fusion methods in diverse applications.

Introduction

Accurate road network data is fundamental to modern navigation, urban planning, and disaster response. Traditionally, these maps are generated through manual digitization or by processing expensive high-resolution satellite imagery. While free satellite data like Sentinel-2 is widely available, its 10m/pixel resolution is often insufficient for precise road delineation, particularly in dense urban environments where roads are narrow or obscured.

This research addresses this "resolution gap" by investigating multi-modal data fusion. Specifically, the study explores whether combining low-resolution imagery with GPS trajectory data can produce high-quality road maps without the high cost of premium satellite data. While GPS data has been combined with high-resolution imagery in previous studies, its synergy with low-resolution, freely available data represents a novel frontier in remote sensing.

Methodology

The researchers treated road detection as a semantic segmentation task, evaluating three popular U-shaped deep learning architectures: U-Net, ResUnet (incorporating residual units to mitigate vanishing gradients), and D-Linknet (utilizing dilated convolutions to capture broader spatial context).

Fusion Strategies

A core contribution of this work is the systematic evaluation of different fusion strategies to integrate satellite and GPS data:

  • Early Fusion: Concatenating satellite imagery and GPS trajectory data at the input level before feeding them into a single model stream.
  • Late Fusion Type-1: Processing only the satellite stream through the model and merging the resulting features with raw GPS data at the final stages.
  • Late Fusion Type-2: Employing separate model streams for both satellite and GPS data, then merging their high-level features before the final output layer.
Architectural diagrams of the four fusion strategies evaluated in the study: (a) baseline satellite-only, (b) early fusion, (c) late fusion type-1, and (d) late fusion type-2.

Experiments also involved testing various fusion operators (Concatenate, Average, Maximum, and Multiply) and loss functions (MSE, Binary Cross-Entropy, and Focal Loss) to optimize the handling of class imbalance, as roads typically occupy a small fraction of the total image area.

Datasets

The study utilized data from two distinct geographic domains to test model generalizability:

  1. Istanbul, Turkey: A large-scale dataset containing approximately 360 million GPS points from diverse vehicle types.
  2. Montreal, Canada: A dataset comprising roughly 40 million GPS points from passenger cars.
  3. Sentinel-2 Satellite Imagery: 10m resolution multi-spectral data (RGB + Infrared), upscaled to 2.5m for processing.
  4. OpenStreetMap (OSM): Used as the ground truth labels, with road widths adjusted based on functional classification.

Key Insights and Results

GPS as a Structural Skeleton

GPS trajectories provide a precise skeleton of human movement that compensates for the lack of visual detail in low-resolution imagery. This multi-modal approach allows the model to "see" roads that might be obscured by shadows, clouds, or vegetation in the satellite view.

Performance Benchmarks

  • Top Performer: The ResUnet model using Late Fusion Type-2 (with concatenation and MSE loss) achieved the highest performance.
  • Quantitative Gains: This configuration reached a mean Intersection over Union (mIoU) of 0.784, significantly outperforming previous benchmarks that relied on Sentinel-2 data alone (which typically achieved mIoU scores around 0.68–0.70).
  • Late vs. Early Fusion: Late fusion (merging features deep in the network) generally outperformed early fusion, suggesting that the model benefits from learning the unique features of imagery and trajectories independently before integration.

Evaluation Metrics

The authors advocate for the use of Boundary-IoU alongside standard IoU. While standard IoU can be high if a model predicts a general "blob" over a road, Boundary-IoU specifically measures how well the edges of the predicted road align with the ground truth, which is critical for practical navigation applications.

Limitations and Future Work

Despite the success of the fusion models, the study noted a "Complexity Gap." Istanbul’s urban fabric is significantly more complex than Montreal’s, and models trained on one city did not generalize perfectly to the other (showing a drop of approximately 0.2 mIoU). This indicates that local urban characteristics—such as building density and road materials—remain a challenge for global road mapping.

Future research directions include exploring Transformer-based architectures (such as SegFormer) and incorporating more diverse regularization strategies to improve cross-city generalization. Additionally, while Late Fusion Type-2 is highly effective, it increases computational costs by requiring parallel networks, a factor that must be balanced in large-scale deployments.

Summary updated

Abstract

This study presents an innovative approach for automatic road detection with deep learning, by employing fusion strategies for utilizing both lower-resolution satellite imagery and GPS trajectory data, a concept never explored before. We rigorously investigate both early and late fusion strategies, and assess deep learning based road detection performance using different fusion settings. Our extensive ablation studies assess the efficacy of our framework under diverse model architectures, loss functions, and geographic domains (Istanbul and Montreal). For an unbiased and complete evaluation of road detection results, we use both region-based and boundary-based evaluation metrics for road segmentation. The outcomes reveal that the ResUnet model outperforms U-Net and D-Linknet in road extraction tasks, achieving superior results over the benchmark study using low-resolution Sentinel-2 data. This research not only contributes to the field of automatic road detection but also offers novel insights into the utilization of data fusion methods in diverse applications.
BibTeX
@inproceedings{gengec2024ai,
  author = {Necip Enes Gengeç and Ergin Tari and Ulas Bagci},
  title = {{AI Powered Road Network Prediction with Multi-Modal Data}},
  booktitle = {Earth Sci. Informatics},
  year = {2024},
  url = {https://openreview.net/forum?id=IOkxeHADz8},
}
Figure