High-Resolution Aerial Imagery Semantic Labeling with Dense Pyramid Network

Xuran Pan, Lianru Gao, Bing Zhang, Fan Yang, Wenzhi LiaoSensors 2018 · OpenReview

TL;DR

This paper introduces a Dense Pyramid Network (DPN) for semantic segmentation of high-resolution aerial imagery, tackling challenges like intra-class variance and inter-class similarity. The DPN leverages group convolutions, channel shuffling, dense blocks, and a pyramid pooling module for effective multi-sensor and multi-resolution feature fusion, achieving state-of-the-art results.

What Problem Does This Paper Address?

High-resolution aerial imagery is crucial for applications like urban planning and environmental monitoring. However, its increased detail leads to significant challenges for semantic segmentation:

  • Intra-class variance: Objects within the same class can look very different (e.g., various types of buildings, cars of different colors).
  • Inter-class similarity: Objects from different classes can appear very similar (e.g., roads and sidewalks, different types of vegetation).
  • Class imbalance: Some classes (like cars) are much rarer than others (like impervious surfaces), making it hard for models to learn them effectively.

Existing deep convolutional neural networks (DCNNs) struggle to capture the fine-grained details and contextual information needed to overcome these ambiguities, especially when fusing data from multiple sensors (like RGB imagery and Digital Surface Models).

Methods

The proposed Dense Pyramid Network (DPN) is designed to address these challenges through several key components:

  • Group Convolutions and Channel Shuffling: To effectively utilize multi-sensor data, the network first applies group convolutions and channel shuffling. This approach processes each sensor's data channel-wise, preserving more information and enhancing the flow of information between channels compared to simply stacking them as input.
  • Dense Convolutional Blocks: Four densely connected convolutional blocks are employed for high-level feature extraction. These blocks connect each layer to every other layer in a feed-forward fashion, which helps mitigate the vanishing gradient problem, promotes feature reuse, and strengthens feature propagation.
  • Pyramid Pooling Module (PPM): To fuse multi-resolution features and capture global contextual information, a pyramid pooling module is integrated. This module uses multi-scale pooling and up-sampling layers to aggregate features from different receptive fields, effectively creating a global scenery prior. This helps to balance the trade-off between recognition and localization.
  • Median Frequency Balanced Focal Loss (MFFL): To combat the class imbalance problem, the authors propose a modified focal loss. This loss function down-weights the contribution of well-classified samples and focuses more on hard-to-classify examples. They further constrain the median frequency weights to prevent overreaction to specific classes, particularly small objects like cars.

Datasets

The proposed DPN was evaluated on two widely used high-resolution aerial imagery datasets:

  • ISPRS Vaihingen Dataset: Contains 33 true orthophoto tiles with RGB and Near-Infrared (NIR) channels, along with Digital Surface Models (DSMs) and normalized DSMs (nDSMs). It is labeled into six classes: impervious surfaces, building, low vegetation, tree, car, and clutter.
  • ISPRS Potsdam Dataset: Consists of 38 true orthophoto tiles with RGB, NIR, and Blue channels, plus DSMs and nDSMs. It is also labeled into the same six classes as the Vaihingen dataset.

Key Insights and Ideas

  • Channel-wise processing for multi-sensor data is crucial: Simply stacking multi-sensor data at the input layer can lead to information loss and classification ambiguity. The proposed group convolution and channel shuffling approach effectively preserves and integrates information from different sensors.
  • Dense connections enhance feature representation: The dense blocks allow for better feature propagation and reuse, leading to richer feature representations that are essential for distinguishing subtle differences in high-resolution imagery.
  • Pyramid pooling captures global context effectively: The PPM's ability to aggregate features at multiple scales provides a global contextual understanding, which is vital for resolving ambiguities and improving segmentation accuracy, especially in challenging scenarios like shadows.
  • A tailored loss function is key for imbalanced datasets: The median frequency balanced focal loss significantly improves the segmentation of small, underrepresented classes (like cars) by focusing the learning process on difficult examples.

Key Results

The proposed DPN, particularly when trained with the MFFL (DPN-MFFL), demonstrated superior performance on both the Vaihingen and Potsdam datasets compared to state-of-the-art methods.

  • Vaihingen Dataset: The DPN-MFFL achieved an overall accuracy (OA) of 89.26% and an average F1-score of 88.36%. Notably, it significantly improved the F1-score for the "car" class to 85.69%, a substantial gain over other methods.
  • Potsdam Dataset: The DPN-MFFL achieved an OA of 90.4% and an average F1-score of 92.1%. It outperformed other methods across most classes, showing strong performance even on complex patterns and challenging objects.
  • Qualitative Results: Visual comparisons showed that the DPN-MFFL was more robust to shadows, better distinguished between similar classes, and more accurately segmented small objects like cars, providing finer-grained segmentation maps with better visual quality.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the DPN-MFFL shows significant improvements, there is still room for enhancement, particularly in handling extremely complex clutter classes or very challenging shadow conditions. The computational cost of dense blocks and pyramid pooling, while effective, can be a consideration for real-time applications.
  • Future Work: The authors suggest that further exploration of multi-sensor fusion strategies and more advanced loss functions could lead to even better performance. Investigating lightweight versions of the DPN for resource-constrained environments could also be a promising direction.

Summary updated

Abstract

Semantic segmentation of high-resolution aerial images is of great importance in certain fields, but the increasing spatial resolution brings large intra-class variance and small inter-class differences that can lead to classification ambiguities. Based on high-level contextual features, the deep convolutional neural network (DCNN) is an effective method to deal with semantic segmentation of high-resolution aerial imagery. In this work, a novel dense pyramid network (DPN) is proposed for semantic segmentation. The network starts with group convolutions to deal with multi-sensor data in channel wise to extract feature maps of each channel separately; by doing so, more information from each channel can be preserved. This process is followed by the channel shuffle operation to enhance the representation ability of the network. Then, four densely connected convolutional blocks are utilized to both extract and take full advantage of features. The pyramid pooling module combined with two convolutional layers are set to fuse multi-resolution and multi-sensor features through an effective global scenery prior manner, producing the probability graph for each class. Moreover, the median frequency balanced focal loss is proposed to replace the standard cross entropy loss in the training phase to deal with the class imbalance problem. We evaluate the dense pyramid network on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam 2D semantic labeling dataset, and the results demonstrate that the proposed framework exhibits better performances, compared to the state of the art baseline.
BibTeX
@inproceedings{pan2018high,
  author = {Xuran Pan and Lianru Gao and Bing Zhang and Fan Yang and Wenzhi Liao},
  title = {{High-Resolution Aerial Imagery Semantic Labeling with Dense Pyramid Network}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=0dGTPErl2Z},
}
Figure