Active Learning for Improved Semi-Supervised Semantic Segmentation in Satellite Images

Shasvat Desai, Debasmita GhoseWACV 2022

TL;DR

This paper tackles the challenge of limited labeled data for satellite image semantic segmentation by proposing an active learning strategy to intelligently select the most informative images for pixel-level annotation. This approach significantly improves the performance of semi-supervised segmentation models, achieving up to a 27% mIoU improvement with as little as 2% labeled data compared to random sampling.

What Problem Does This Paper Address?

Accurate semantic segmentation of satellite imagery is crucial for a wide range of applications, from environmental monitoring to urban planning. However, creating dense, pixel-level annotations for these images is a labor-intensive and expensive process. This scarcity of labeled data is a major bottleneck. While semi-supervised learning (SSL) offers a solution by leveraging unlabeled data alongside a small labeled set, the performance of SSL heavily relies on the quality and representativeness of the initial labeled data. Randomly selecting this small subset can lead to biased models that underperform. This paper addresses this by proposing a method to select a highly representative and diverse set of labeled training data using active learning.

Methods

The core of this work lies in an active learning (AL) strategy designed to select informative images for pixel-level annotation, rather than just individual pixels or patches. The authors employ a pool-based active learning approach, where a learner (an image classifier) is iteratively trained on a growing set of labeled data.

Here's a breakdown of their method:

  • Active Learning for Image Classification: They use a pre-trained image classification network (ResNet 101/50) as their "learner." This learner is trained on an initial small set of labeled images.
  • Query Strategies: Two common AL query strategies are explored:
  • Entropy-based sampling: Selects images where the classifier is most uncertain about the predicted class.
  • Margin sampling: Selects images where the classifier's top two predicted class probabilities are closest.
  • Image-Level Selection: Instead of selecting individual pixels, the AL strategy identifies entire images from an unlabeled pool that are deemed most informative based on the chosen query strategy.
  • Oracle for Pixel-Level Labels: Once an image is selected by the AL module, a human annotator (the "oracle") provides the dense pixel-level ground truth labels for that specific image.
  • Semi-Supervised Semantic Segmentation (s4GAN): The selected images and their pixel-level labels are then used to train a conditional Generative Adversarial Network (cGAN) for semantic segmentation. The cGAN, based on Mittal et al. [31], uses a generator (a segmentation network) and a discriminator. It leverages both labeled and unlabeled data to generate pseudo-labels for further training.

The novelty lies in using an image-classification-based active learning approach to intelligently curate the initial labeled dataset for a semi-supervised semantic segmentation task, specifically for satellite imagery.

Datasets

The proposed method was evaluated on two publicly available satellite image datasets:

  • UC Merced Land Use Classification Dataset: Contains 2100 RGB images (256x256 pixels) with 21 land-use classes. The authors used pixel-level annotations from Shao et al. [46], which provides 17 classes.
  • DeepGlobe Land Cover Classification Dataset: Consists of DigitalGlobe Vivid+ images (2448x2448 pixels) with 7 land-cover classes. For this dataset, the authors generated image-level annotations by assigning the class with the highest pixel count to each image.

Key Insights and Ideas

  • Representativeness is Key: The core insight is that the quality and diversity of the initial labeled data are far more critical for semi-supervised learning than just the sheer quantity, especially in low-data regimes.
  • Active Learning for Image Selection: Applying active learning at the image level, rather than just pixel or patch level, is an effective way to guide the expensive pixel-level annotation process.
  • Bridging Image Classification and Semantic Segmentation: The paper successfully bridges the gap between image-level classification tasks (where AL is common) and dense pixel-level prediction tasks (semantic segmentation) by using AL to inform the latter.
  • Significant Gains with Minimal Labels: The results demonstrate that even with a very small fraction of labeled data (as low as 2%), an intelligently selected set can yield performance comparable to or even exceeding models trained on much larger, randomly sampled datasets.

Key Results

The paper reports significant improvements in mean Intersection-over-Union (mIoU) scores:

  • UC Merced Dataset: Achieved up to a 15% mIoU improvement with 2% labeled data and up to an 18% improvement with 5% and 12.5% labeled data, compared to the baseline random sampling approach.
  • DeepGlobe Dataset: Demonstrated a maximum 27% mIoU improvement with 2% labeled data, and approximately 6-8% improvement with 5% and 12.5% labeled data.
  • Qualitative Results: Figures 3 and 4 show that the proposed method produces significantly cleaner segmentation boundaries and fewer false positives, especially in challenging scenarios like distinguishing coastlines or complex shapes, even with very few labeled examples.

Limitations and Future Work

  • Computational Cost of Active Learning: While effective, the active learning process itself can be computationally intensive, requiring iterative training of the image classifier.
  • Oracle Dependency: The effectiveness of the method relies on the availability of an "oracle" that can provide accurate pixel-level labels for the selected images.
  • Future Work: The authors suggest that this active learning approach can serve as a guiding process for large-scale dataset creation, making the process of obtaining dense pixel-level annotations more cost-effective by first focusing on image-level selection. Further exploration of different active learning strategies and their impact on various satellite image datasets could be beneficial.

Summary updated

BibTeX
@inproceedings{desai2022active,
  author = {Shasvat Desai and Debasmita Ghose},
  title = {{Active Learning for Improved Semi-Supervised Semantic Segmentation in Satellite Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure