Semantic Segmentation of Remote-Sensing Imagery Using Heterogeneous Big Data: International Society for Photogrammetry and Remote Sensing Potsdam and Cityscape Datasets

Ahram Song, Yongil KimISPRS Int. J. Geo Inf 2020 · DOI

TL;DR

This paper tackles the challenge of limited labeled remote-sensing (RS) data for semantic segmentation by proposing a novel "Combined U-Net" model. This model leverages a large, heterogeneous natural image dataset (Cityscapes) to improve segmentation accuracy on smaller RS datasets (ISPRS Potsdam), demonstrating that cross-domain learning can effectively overcome data scarcity in RS applications.

What Problem Does This Paper Address?

Semantic segmentation, a crucial task for understanding images by assigning a label to each pixel, has seen significant advancements with deep learning. However, applying these methods to remote-sensing (RS) imagery faces a major hurdle: the scarcity of large, meticulously labeled datasets. Unlike natural images, acquiring RS data under consistent conditions and manually annotating them is extremely difficult and time-consuming. This data limitation hinders the effective training of deep learning models, leading to suboptimal performance. While pre-training on natural image datasets like ImageNet is common, directly learning from heterogeneous RS and natural image datasets simultaneously to boost RS segmentation accuracy remains an underexplored area.

Methods

The core of this paper is the Proposed Combined U-Net Model. This architecture builds upon the U-Net, a popular choice for image segmentation, but is adapted for heterogeneous data. The key innovation lies in its shared encoder blocks and separate decoder blocks.

  • Shared Encoder: The initial encoder layers are shared between the RS (ISPRS Potsdam) and natural image (Cityscapes) data streams. This allows the model to learn common, low-level features that are transferable across different image domains.
  • Separate Decoders: After the shared encoding, each data stream has its own decoder. This enables the model to learn domain-specific features and perform segmentation tailored to each dataset.
  • Combined Weighted Loss Function: The model is trained using a weighted sum of the loss functions from both the RS and natural image decoders. The weights ($w_1$ and $w_2$) are empirically determined, with a higher weight typically assigned to the RS data loss to prioritize improving RS segmentation accuracy.

This approach aims to leverage the abundance of labeled data in natural image datasets to compensate for the lack of labeled data in RS datasets, thereby improving the overall segmentation performance for RS imagery.

Datasets

The study utilizes two distinct datasets:

  • ISPRS Potsdam/Germany Dataset: This is the remote-sensing (RS) dataset. It consists of high-resolution (5 cm) airborne images with near-infrared (NIR), red, blue, and green bands, along with corresponding ground-truth semantic labels. For training and validation, 24 patches were used.
  • Cityscapes Dataset: This is the natural image dataset. It comprises stereo video sequences from urban street scenes, featuring dense pixel-level labeling for over 30 classes. This dataset is significantly larger than the ISPRS Potsdam dataset.

The authors note that for the combined model to effectively share encoder blocks, only the visible bands (RGB) of the ISPRS Potsdam dataset were used, aligning with the bands available in the Cityscapes dataset.

Key Insights and Ideas

  • Cross-Domain Learning for RS Data Scarcity: The most significant insight is that leveraging large natural image datasets can indeed improve semantic segmentation performance on smaller, specialized RS datasets, directly addressing the data scarcity problem.
  • Shared Encoders are Key: Sharing the initial encoder layers is crucial for learning transferable features, allowing the model to benefit from the diverse information present in both datasets without requiring identical data structures.
  • Dataset Size Matters: Increasing the size of the natural image dataset (Cityscapes) significantly improved the performance of the combined model, highlighting the importance of abundant data for effective cross-domain learning.
  • NIR Band Importance: The study suggests that while using RGB bands from RS data is necessary for cross-domain learning with natural images, the NIR band is vital for accurately classifying vegetation-related classes, and its exclusion can lead to performance degradation in those specific classes.

Key Results

  • Improved RS Segmentation: The proposed Combined U-Net model, particularly when trained with a larger Cityscapes dataset (Case 2), achieved a higher overall accuracy (OA) on the ISPRS Potsdam test set (0.8721) compared to single models like SegNet (0.8346) and simplified U-net (0.7841), and even surpassed DeepLab-V3+ (0.8605).
  • Reduced Overfitting: Increasing the size of the Cityscapes dataset helped to mitigate the overfitting problem, leading to better generalization on the RS data.
  • Impact of Dataset Weights: The study confirmed that the weighting of the loss functions ($w_1$ and $w_2$) plays a role. Assigning a higher weight to the RS data loss ($w_1=0.8, w_2=0.2$) yielded better results than a balanced weighting ($w_1=0.2, w_2=0.8$).
  • Qualitative Improvements: Visual analysis showed that Case 2 of the combined U-net model was more effective in classifying buildings and roads, and better predicted road boundaries and car shapes compared to other methods.

Limitations and Future Work

  • Computational Burden: The combined U-net model is computationally more intensive and takes longer to train compared to single-dataset models due to processing heterogeneous data.
  • Memory Constraints: The authors experienced memory issues when using batch sizes larger than 4 on Google Colab, indicating potential scalability challenges.
  • NIR Band Exclusion: The necessity of using only RGB bands from RS data for cross-domain learning with natural images limits the model's ability to fully leverage all available spectral information, potentially impacting the accuracy of vegetation-related classes.
  • Future Work: The authors plan to explore including the NIR band information in future work to improve vegetation classification. They also intend to investigate how changing the shared and non-shared parts of the encoder/decoder architecture affects performance. Additionally, they suggest that this approach could be extended to other applications involving heterogeneous datasets, such as floorplan analysis or street-view image segmentation.

Summary updated

BibTeX
@article{song2020semantic,
  author = {Ahram Song and Yongil Kim},
  title = {{Semantic Segmentation of Remote-Sensing Imagery Using Heterogeneous Big Data: International Society for Photogrammetry and Remote Sensing Potsdam and Cityscape Datasets}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2020},
  doi = {10.3390/ijgi9100601},
}
Figure