Prototypical Contrastive Network for Imbalanced Aerial Image Segmentation

Keiller Nogueira, Mayara Maezano Faita-Pinheiro, Ana Paula Marques Ramos, Wesley Nunes Gonçalves, José Marcato Junior, Jefersson A. dos SantosWACV 2024

TL;DR

This paper introduces a novel deep learning approach for imbalanced binary segmentation of remote sensing images. By focusing on learning representations for the foreground class and using a contrastive loss that pushes difficult background samples away from foreground prototypes, the method achieves state-of-the-art performance with significantly reduced training time.

What Problem Does This Paper Address?

Binary segmentation is crucial for many remote sensing applications, such as identifying specific objects like cars or water bodies. However, this task is plagued by two major challenges: huge intra-class variance within the background (e.g., forests, cities, roads all being "background") and extreme class imbalance, where the background class vastly outnumbers the foreground. Traditional methods often struggle with these issues, leading to biased models that perform poorly on the minority foreground class. Existing approaches that try to address imbalance often focus on learning representations for all classes, which can be difficult given the background's diversity, or they rely on complex hard-example mining techniques that increase training time.

Methods

The core of the proposed method, the "Prototypical Contrastive Network," lies in its unique training strategy. Instead of learning prototypes for all classes, it learns a single prototype (or a few prototypes) specifically for the foreground class. A fully convolutional backbone extracts pixel-wise features, which are then normalized. The key innovation is a novel contrastive loss function that:

  1. Pulls foreground samples closer to their learned prototype.
  2. Pushes background samples away from the foreground prototype.

Crucially, this loss function implicitly focuses on the most difficult background samples. By only considering background pixels whose feature representations are close to the foreground prototype (within a defined margin), the model is forced to learn to distinguish these challenging negative examples. This approach alleviates the need to learn comprehensive representations for the entire diverse background, thereby simplifying the learning process and reducing training time.

Datasets

The authors evaluated their method on two datasets:

  • Vaihingen: A publicly available, high-resolution remote sensing dataset. For this work, the "car" class was treated as the foreground, creating an imbalanced scenario.
  • River: A new dataset introduced by the authors, specifically designed to present an extreme class imbalance scenario. It consists of three high-resolution RGB orthoimages annotated for water (foreground) and non-water (background) classes, totaling over 500 million pixels.

Key Insights and Ideas

  • Focusing on the Positive: The most significant insight is that by learning representations only for the foreground class and using contrastive learning to push away difficult background samples, the model can achieve better performance and efficiency. This shifts the paradigm from trying to understand everything to learning to distinguish a specific target.
  • Implicit Hard-Example Mining: The contrastive loss, by its nature, automatically identifies and focuses on the most challenging background samples (those closest to the foreground prototype). This is a more elegant and efficient way to handle imbalance than explicit hard-example mining techniques.
  • Computational Efficiency: The simplified learning objective and implicit mining lead to substantially reduced training times, a critical factor for large-scale remote sensing datasets.
  • Single Prototype Sufficiency: Experiments showed that using just one prototype for the foreground class yielded the best results, suggesting that complex prototype structures are not always necessary for effective segmentation in imbalanced scenarios.

Key Results

The proposed Prototypical Contrastive Network demonstrated strong performance:

  • State-of-the-Art Performance: It achieved the best F1-scores on both the Vaihingen (79.07%) and River (78.78%) datasets, outperforming several established baseline methods.
  • Computational Efficiency: The method achieved these results with significantly less training time, taking less than half the time of most competing approaches. For instance, on the River dataset, it trained in 66.6 hours, while baselines took over 120 hours.
  • Qualitative Improvements: Visualizations showed that the proposed method effectively groups foreground samples around the prototype and pushes away negative instances, leading to cleaner segmentation boundaries.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance is influenced by hyperparameters like the margin ($m$). The authors performed margin analysis and found optimal values of 4 for Vaihingen and 3 for River, suggesting that careful tuning might be necessary.
  • Future Directions: The authors plan to explore more advanced backbones (e.g., Vision Transformers, Swin Transformers) and investigate different reduction strategies for the loss function (e.g., mean vs. sum). They also aim to test the approach on a wider variety of datasets and applications.

Summary updated

BibTeX
@inproceedings{nogueira2024prototypical,
  author = {Keiller Nogueira and Mayara Maezano Faita-Pinheiro and Ana Paula Marques Ramos and Wesley Nunes Gonçalves and José Marcato Junior and Jefersson A. dos Santos},
  title = {{Prototypical Contrastive Network for Imbalanced Aerial Image Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure