Semantic segmentation of remote sensing imagery for road extraction via joint angle prediction: comparisons to deep learning

Shun Xiong, Chao Ma, Guang Yang, Yaodong Song, Shuaizhe Liang, Jing FengFrontiers in Earth Science 2023 · DOI

TL;DR

Researchers have developed a new deep learning approach for road extraction from satellite imagery that solves the common problem of "broken" or jagged road networks. By training a model to simultaneously predict road segments and their local orientation (angles), the system uses geometric constraints to produce smoother, more continuous, and map-ready road data.

What Problem Does This Paper Address?

Extracting road networks from remote sensing imagery is a cornerstone of modern cartography and urban planning. While deep learning—specifically semantic segmentation—has made huge strides, it still struggles with the unique geometry of roads.

Standard models often treat road extraction as a simple pixel-classification task ("is this pixel a road or not?"). Because roads are thin, long, and often obscured by shadows or trees, these models frequently produce "fractured" results: roads that have gaps, inconsistent widths, or "noisy" edges. These artifacts make the data difficult to use for actual navigation or urban subsidence studies without extensive manual cleanup. The authors argue that traditional methods lack geometric constraints—the inherent understanding that a road should follow a consistent path and angle.

Methods

The authors propose a novel architecture that moves beyond simple classification by incorporating joint angle prediction.

  1. Backbone (Swin Transformer): Instead of a traditional CNN, the model uses a Swin Transformer. This allows the network to capture long-range dependencies in the image, which is vital for recognizing long, linear features like highways.
  2. Dual-Branch Decoding: The features from the backbone are sent into two parallel modules:
  3. Semantic Segmentation Module: Makes a preliminary prediction of where the road is.
  4. Angle Prediction Module: Predicts the specific inclination angle of the road at every pixel (constrained between $-\pi/2$ and $\pi/2$).
  5. Angle Feature Fusion: This is the "aha" component. The model takes the predicted angles and fuses them back into the segmentation features. This acts as a geometric filter, ensuring that the predicted road pixels align with the predicted road direction.
  6. Multi-Task Loss Function: The model is trained using a composite loss function that penalizes errors in the preliminary mask, the predicted angles, and the final fused road extraction result.

Datasets

The study utilizes the DeepGlobe public dataset, a benchmark in the remote sensing community.

  • Specs: 6,226 training images at 0.5m spatial resolution.
  • Preprocessing: The original $1024 \times 1024$ images were re-split into $512 \times 512$ blocks to optimize training.
  • Comparison: The authors compared their results against a wide array of state-of-the-art models, including DeepLabv3+, BRRNet, and Mask2Former.

Key Insights and Ideas

  • Angle as a Prior: The most significant insight is that road direction is a stable geometric property. By forcing the model to learn "which way the road points," the network learns to ignore visual noise (like a car or a small shadow) that doesn't align with that direction.
  • Transformers for Linear Features: The shift to a Swin Transformer backbone is crucial. Because roads span large portions of an image, the "global" view of a Transformer is better suited for this task than the "local" view of standard convolutional kernels.
  • Connectivity over Pixel Accuracy: While standard metrics like IoU are important, the authors emphasize that the continuity of the road is what matters for real-world applications. Their fusion module specifically targets the "fracture" problem.

Key Results

The proposed method outperformed all baseline models on the DeepGlobe dataset:

  • Quantitative: The model achieved an IoU (Intersection over Union) of 0.7132 and an F1 score of 0.8323, surpassing high-performing models like GCB-Net and Mask2Former.
  • Qualitative: Visual inspections (see Figure 4 in the paper) show that while other models produced fragmented road segments in complex urban or rural areas, the proposed method maintained consecutive, distinct edges.
  • Efficiency: Despite the added complexity of the angle module, the parameter count (117.12 million) remained comparable to the standard Swin Transformer, suggesting the performance boost comes from better architecture design rather than just "throwing more parameters" at the problem.

Limitations and Future Work

The authors identify several areas for improvement:

  • Real-time Performance: While accurate, the training and inference times for Transformer-based models are still relatively high compared to lightweight CNNs.
  • Data Fusion: The current model relies solely on RGB imagery. The authors suggest that incorporating Lidar or other geographic information systems (GIS) data could further improve robustness in heavily forested or shadowed areas.
  • Large-Scale Deployment: The model has been tested on benchmark datasets, but its performance in "wild" large-scale urban environments with varying regional road styles remains a target for future research.

Summary updated

Abstract

Accurate road network information is required to study and analyze the relationship between land usage type and land subsidence, and road extraction from remote sensing images is an important data source for updating road networks. This task has been considered a significant semantic segmentation problem, given the many road extraction methods developed for remote sensing images in recent years. Although impressive results have been achieved by classifying each pixel in the remote sensing image using a semantic segmentation network, traditional semantic segmentation methods often lack clear constraints of road features. Consequently, the geometric features of the results might deviate from actual roads, leading to issues like road fractures, rough edges, inconsistent road widths, and more, which hinder their effectiveness in road updates. This paper proposes a novel road semantic segmentation algorithm for remote sensing images based on the joint road angle prediction. By incorporating the angle prediction module and the angle feature fusion module, constraints are added to the angle features of the road. Through the angle prediction and angle feature fusion, the information contained in the remote sensing images can be better utilized. The experimental results show that the proposed method outperforms existing semantic segmentation methods in both quantitative evaluation and visual effects. Furthermore, the extracted roads were consecutive with distinct edges, making them more suitable for mapping road updates.
BibTeX
@inproceedings{xiong2023semantic,
  author = {Shun Xiong and Chao Ma and Guang Yang and Yaodong Song and Shuaizhe Liang and Jing Feng},
  title = {{Semantic segmentation of remote sensing imagery for road extraction via joint angle prediction: comparisons to deep learning}},
  booktitle = {Frontiers in Earth Science},
  year = {2023},
  doi = {10.3389/feart.2023.1301281},
}
Figure