High-Resolution Remote Sensing Imagery Classification of Imbalanced Data Using Multistage Sampling Method and Deep Neural Networks

Wei Xia, Caihong Ma, Jianbo Liu, Shibin Liu, Chen Fu, Zhi Yang, Jianbo DuanRemote 2019 · DOI

TL;DR

This paper tackles the critical problem of class imbalance in high-resolution remote sensing image classification, where minority classes (like roads or buildings) are often poorly detected. The authors propose a novel approach combining a sophisticated deep learning architecture (Encoder-Decoder with ASPP) with a "multistage sampling" technique to create more balanced training datasets, leading to significantly improved classification accuracy for these challenging minority classes.

What Problem Does This Paper Address?

High-resolution remote sensing images are incredibly valuable for detailed land cover monitoring, but their classification using deep learning is hampered by a common issue: class imbalance. This means that some ground object classes (e.g., vast areas of bare land or forest) are vastly overrepresented in training data compared to others (e.g., narrow roads or small buildings). When deep learning models are trained on such imbalanced data, they tend to become biased towards the majority classes, resulting in poor performance and low accuracy for the underrepresented minority classes. This significantly limits the practical utility of these powerful classification models in real-world applications.

Methods

The core of the proposed solution lies in two key components:

  • Encoder-Decoder with Atrous Spatial Pyramid Pooling (ASPP): This is a state-of-the-art deep learning architecture for pixel-wise image segmentation. The "Encoder" part uses atrous convolutions (which allow for larger receptive fields without losing resolution) and ASPP to capture multi-scale contextual information. The "Decoder" part then uses this information to reconstruct the spatial details and restore the resolution, enabling precise pixel-level classification. This architecture is designed to handle the rich details present in high-resolution imagery.
  • Multistage Sampling: To address class imbalance, the authors introduce a novel "multistage sampling" technique. This involves two stages (MUS1 and MUS2) of hybrid sampling. Instead of simply over- or under-sampling, this method aims to create more balanced training sets by carefully adjusting the proportions of samples from different classes. The MUS2 stage, in particular, is designed to achieve a finer balance, leading to improved performance for minority classes.

Datasets

The study utilizes two distinct datasets:

  • WorldView-3 Images: High-resolution (0.31m panchromatic, 1.24m multispectral) images from the southeastern Qinghai-Tibet Plateau. This dataset features complex landforms, uneven distribution of ground objects, and small target sizes, making it a challenging testbed for imbalanced classification.
  • GF-2 Images: High-resolution (0.8m) true color fusion images from northeastern Beijing, representing a dense urban area. This dataset also exhibits significant class imbalance, particularly in urban scenarios.

The paper also refers to existing benchmarks like the UC Merced Land Use dataset, but the primary focus is on the custom-labeled datasets collected for this study.

Key Insights and Ideas

  • Multistage sampling is crucial: Simply using original imbalanced datasets or even basic sampling methods (like random over/under-sampling) is insufficient for achieving high accuracy on minority classes. The proposed multistage sampling, especially the MUS2 stage, demonstrably improves performance.
  • Encoder-Decoder with ASPP excels at detail: The chosen deep learning architecture is well-suited for high-resolution imagery, effectively capturing both contextual information and fine spatial details, which are essential for accurate pixel-level classification.
  • Minority class performance is the real test: Overall accuracy can be misleading with imbalanced data. Metrics like G-mean and per-class recall/precision are vital for understanding how well the model performs on the underrepresented classes.
  • Combining sampling and architecture matters: The success of the approach hinges on both the sophisticated network architecture and the carefully crafted training data generated by multistage sampling.

Key Results

  • The proposed approach (Encoder-Decoder with ASPP and MUS2 sampling) achieved the highest Overall Accuracy (OA) and G-mean values on both the Mangkam (WorldView-3) and Northeastern Beijing (GF-2) datasets compared to other methods.
  • For the Mangkam dataset, the MUS2 sampling resulted in an OA of 0.88 and G-mean of 0.84, representing significant improvements over the original dataset.
  • Crucially, the classification accuracies for minority classes like roads and buildings were substantially enhanced. For instance, road identification accuracy improved by 0.53 and building identification by 0.52 on the Mangkam dataset.
  • Visual comparisons showed that the proposed method produced classification maps with more accurate boundaries and fewer misclassifications, especially for roads and buildings, compared to basic FCN, FCN-8s, and ASPP models.

Limitations and Future Work

  • Data Annotation Effort: The paper acknowledges that obtaining high-quality, labeled remote sensing data is labor-intensive and depends heavily on the experience of classification personnel.
  • Sample Size: While the multistage sampling improves balance, the absolute number of annotated samples can still be a limiting factor for deep learning models.
  • Future Work: The authors suggest that semi-supervised learning methods could be explored to reduce the reliance on extensive manual labeling and to investigate other remote sensing data sources. They also propose that their framework has the potential to adapt to a wider range of engineering and multi-source remote sensing satellite data.

Summary updated

Abstract

Class imbalance is a key issue for the application of deep learning for remote sensing image classification because a model generated by imbalanced samples training has low classification accuracy for minority classes. In this study, an accurate classification approach using the multistage sampling method and deep neural networks was proposed to classify imbalanced data. We first balance samples by multistage sampling to obtain the training sets. Then, a state-of-the-art model is adopted by combining the advantages of atrous spatial pyramid pooling (ASPP) and Encoder-Decoder for pixel-wise classification, which are two different types of fully convolutional networks (FCNs) that can obtain contextual information of multiple levels in the Encoder stage. The details and spatial dimensions of targets are restored using such information during the Decoder stage. We employ four deep learning-based classification algorithms (basic FCN, FCN-8S, ASPP, and Encoder-Decoder with ASPP of our approach) on multistage training sets (original, MUS1, and MUS2) of WorldView-3 images in southeastern Qinghai-Tibet Plateau and GF-2 images in northeastern Beijing for comparison. The experiments show that, compared with existing sets (original, MUS1, and identical) and existing method (cost weighting), the MUS2 training set of multistage sampling significantly enhance the classification performance for minority classes. Our approach shows distinct advantages for imbalanced data.
BibTeX
@inproceedings{xia2019high,
  author = {Wei Xia and Caihong Ma and Jianbo Liu and Shibin Liu and Chen Fu and Zhi Yang and Jianbo Duan},
  title = {{High-Resolution Remote Sensing Imagery Classification of Imbalanced Data Using Multistage Sampling Method and Deep Neural Networks}},
  booktitle = {Remote},
  year = {2019},
  doi = {10.3390/rs11212523},
}
Figure