DeepDualMapper: A Gated Fusion Network for Automatic Map Extraction Using Aerial Images and Trajectories

Hao Wu, Hanyuan Zhang, Xinyu Zhang, Weiwei Sun, Baihua Zheng, Yuning Jiang

Abstract

Automatic map extraction is of great importance to urban computing and location-based services. Aerial image and GPS trajectory data refer to two different data sources that could be leveraged to generate the map, although they carry different types of information. Most previous works on data fusion between aerial images and data from auxiliary sensors do not fully utilize the information of both modalities and hence suffer from the issue of information loss. We propose a deep convolutional neural network called DeepDualMapper which fuses the aerial image and trajectory data in a more seamless manner to extract the digital map. We design a gated fusion module to explicitly control the information flows from both modalities in a complementary-aware manner. Moreover, we propose a novel densely supervised refinement decoder to generate the prediction in a coarse-to-fine way. Our comprehensive experiments demonstrate that DeepDualMapper can fuse the information of images and trajectories much more effectively than existing approaches, and is able to generate maps with higher accuracy.

Introduction

Creating and updating city maps is a massive undertaking essential for autonomous driving and urban planning. While we have two primary automated data sources—satellite/aerial imagery and GPS trajectories—neither is perfect.

Aerial images are frequently obscured by "information loss" from tree canopies, building shadows, or clouds. Conversely, GPS trajectories (the digital breadcrumbs left by vehicles) are often sparse or non-existent on minor or private roads. Previous attempts to fuse these two sources typically used "naive" methods: either stacking the raw data at the start (early fusion) or averaging the final predictions (late fusion). These methods fail to account for the fact that at any given pixel, one source might be much more "trustworthy" than the other.

Comparison of aerial imagery, GPS trajectories, and ground truth maps, highlighting issues like tree occlusion and data sparsity.

Methodology

The DeepDualMapper architecture is a dual-stream framework based on the U-Net structure, designed to address the limitations of single-modality extraction. Its core innovation lies in two specific modules:

  1. Gated Fusion Module (GFM): This acts as the network's "decision-maker." Instead of treating both sources equally, the GFM learns to assign weights (gates) to the image and trajectory features. If a road is hidden under a tree in the image but has many GPS pings, the gate will "open" wider for the trajectory data. This gating happens recursively from coarse to fine scales to ensure spatial consistency.
  2. Densely Supervised Refinement Decoder (DSRD): Rather than producing a map in one shot, this decoder uses residual learning to refine the fused features. It cleans up the "noise"—such as isolated pings or jagged edges—to produce a smooth, connected road network.
  3. Shared Prediction & Dense Supervision: To ensure the network compares "apples to apples," the authors use a shared prediction module across all levels of the network. They force the model to try and predict the map from the image alone, the GPS alone, and the fused features simultaneously. This "dense supervision" ensures that both data types are mapped into a consistent, uniform feature space.

Experiments and Datasets

The model was evaluated on three real-world, city-scale datasets:

  • Porto: Known for high-quality satellite imagery and dense trajectory data.
  • Shanghai: Features more complex urban structures and varying trajectory density.
  • Singapore: A challenging environment with significant tropical greenery (causing image occlusion) and diverse road types.

The trajectory data was pre-processed by counting historical GPS points within each pixel grid to create a "heatmap" compatible with standard convolutional layers.

Key Results

DeepDualMapper consistently outperformed all baselines, including specialized road extraction models like DeepRoadMapper and other fusion networks like FuseNet and 1D Decoder.

  • Quantitative Performance: In Porto, the model achieved an Intersection over Union (IoU) of 0.717, compared to 0.642 for a standard U-Net. It showed similar leads in Shanghai and Singapore.
  • Qualitative Improvements: Visual results show that DeepDualMapper is significantly better at connecting broken road segments and ignoring "ghost" roads (noise in GPS data) compared to previous state-of-the-art methods.
  • Robustness: The authors performed "stress tests" by intentionally deleting 25% of the data from both sources. DeepDualMapper maintained high accuracy while other models' performances plummeted, proving that the gated fusion effectively "fills in the blanks."

Conclusion

DeepDualMapper demonstrates that effective data fusion is not just about adding data, but about intelligently selecting the most reliable information at a local level. By using a gated mechanism that mimics human intuition—looking for vehicle movement when visual cues are obscured—the model achieves a new state-of-the-art in automatic map extraction. Future work may involve incorporating richer trajectory data, such as vehicle speed and heading, to further refine the distinction between complex road structures like flyovers and surface roads.

Summary updated

Abstract

Automatic map extraction is of great importance to urban computing and location-based services. Aerial image and GPS trajectory data refer to two different data sources that could be leveraged to generate the map, although they carry different types of information. Most previous works on data fusion between aerial images and data from auxiliary sensors do not fully utilize the information of both modalities and hence suffer from the issue of information loss. We propose a deep convolutional neural network called DeepDualMapper which fuses the aerial image and trajectory data in a more seamless manner to extract the digital map. We design a gated fusion module to explicitly control the information flows from both modalities in a complementary-aware manner. Moreover, we propose a novel densely supervised refinement decoder to generate the prediction in a coarse-to-fine way. Our comprehensive experiments demonstrate that DeepDualMapper can fuse the information of images and trajectories much more effectively than existing approaches, and is able to generate maps with higher accuracy.
BibTeX
@inproceedings{wu2020deepdualmapper,
  author = {Hao Wu and Hanyuan Zhang and Xinyu Zhang and Weiwei Sun and Baihua Zheng and Yuning Jiang},
  title = {{DeepDualMapper: A Gated Fusion Network for Automatic Map Extraction Using Aerial Images and Trajectories}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=aDDN469bJh},
}
Figure