Land Cover Mapping From Multiple Complementary Experts Under Heavy Class Imbalance

Valérie Zermatten, Xiaolong Lu, Javiera Castillo-Navarro, Tobias Kellenberger, Devis TuiaIEEE JSTARS 2024 · DOI · OpenReview

Abstract

In remote sensing, semantic segmentation models often struggle with heavy class imbalance, where common "head" classes dominate rare "tail" classes. This paper introduces the Multiple Complementary Experts (MCE) framework, which utilizes specialized neural network heads to improve the detection of rare land cover types. By training experts on overlapping class subsets and employing a complementary loss function, the MCE framework achieves significant gains in tail class accuracy with minimal impact on overall performance.

Introduction

Semantic segmentation in remote sensing is frequently hindered by the "long-tail" distribution of land cover classes. In mountainous regions, for example, vast areas of forest and bedrock (head classes) often overshadow critical but rare features like glaciers or specific vegetation types (tail classes). Standard deep learning models, acting as "lazy" optimizers, tend to maximize overall accuracy by ignoring these infrequent classes. While various techniques exist for image-level classification imbalance, applying them to pixel-level semantic segmentation is challenging because multiple classes often coexist within a single image.

The Multiple Complementary Experts (MCE) Framework

The authors propose a Multiple Complementary Experts (MCE) architecture to address this imbalance. Rather than a single monolithic network, the framework employs a shared visual backbone that extracts general features, which are then processed by several specialized "expert" heads:

  • Expert 1 (Generalist): Trained across all available classes.
  • Expert 2 (Specialist): Focuses on "body" and "tail" classes (medium to rare frequency).
  • Expert 3 (Niche Specialist): Focuses exclusively on the rarest "tail" classes.
The MCE architecture featuring a shared backbone and specialized expert heads that produce individual logits for aggregation.

To ensure these experts specialize effectively, two key mechanisms are introduced:

  1. Complementary Loss ($L_{com}$): This loss function penalizes experts for predicting classes outside their intended specialization, forcing them to "stay in their lane."
  2. Adaptive Learning Rates: Tail specialists receive lower learning rates to prevent overfitting and the "hallucination" of rare classes in inappropriate contexts.

Datasets and Evaluation

The framework was evaluated on two primary datasets:

  • TLM Dataset: A new, high-resolution alpine dataset covering 2,300 $km^2$ of the Swiss Alps. It contains 10 classes with an extreme distribution where the most common class is 1,000 times more frequent than the rarest.
  • FLAIR Dataset: A large-scale French dataset used to validate the model's generalizability across different geographic regions.

Results and the Aggregation Bottleneck

The MCE approach demonstrated substantial improvements over standard Cross-Entropy baselines. On the TLM dataset using a DeepLabv3+ backbone, accuracy for the rarest classes increased from 23.6% to 62.0%. This massive boost in tail performance came at a minor cost to Overall Accuracy (OA), which only decreased from 89.2% to 87.2%.

A critical finding of the study involves the aggregation module, which combines the expert opinions (logits) into a final prediction. The researchers compared several strategies, including simple averaging and a theoretical "Oracle" model.

Comparison of different logit aggregation strategies, illustrating the gap between automated methods and the theoretical Oracle performance.

The "Oracle" study—which assumes the model always knows which expert to trust for a given pixel—yielded significantly higher results than automated methods. This suggests that while training specialized experts is effective, the current frontier in this research lies in developing more sophisticated, learnable aggregation methods to better synthesize expert wisdom.

Conclusion

The MCE framework provides a robust and relatively simple solution for class imbalance in semantic segmentation. By allowing experts to have overlapping territories, the model fosters complementarity rather than competition. Future work will focus on moving beyond simple logit averaging toward context-aware aggregation sub-networks to further bridge the gap toward Oracle-level performance.

Summary updated

Abstract

Deep learning has emerged as a promising avenue for automatic mapping, demonstrating high efficacy in land cover categorization through various semantic segmentation models. Nonetheless, the practical deployment of these models encounters important challenges from the imbalanced distribution of samples between the classes, a problem inherent to real-world datasets. This results in models biased towards frequent classes that perform poorly on rare classes. While existing approaches to fight class imbalance mainly focus on image classification, here we propose to address this issue for semantic segmentation with a multiple complementary experts (MCE) structure. Taking inspiration from ensemble models, each expert in our MCE specializes in certain classes and works with other experts in a complementary manner to generate robust predictions for rare classes. We compare our approach to other existing methods and also explore different logit aggregation methods, to identify the performance upper bounds and improvement directions. Our model is evaluated on a large-scale and challenging alpine land cover dataset that we make openly available. In addition, we evaluated our model on an imbalanced land cover mapping dataset, FLAIR, to highlight its adaptability. Overall, our MCE model yields notable improvement in performances on the medium and rare classes compared to baseline methods, while only slightly compromising on the overall accuracy. Despite its simplicity, the MCE approach stands as a practical solution for more operational semantic segmentation models, not trading off performances on rare but important classes.
BibTeX
@article{zermatten2024land,
  author = {Valérie Zermatten and Xiaolong Lu and Javiera Castillo-Navarro and Tobias Kellenberger and Devis Tuia},
  title = {{Land Cover Mapping From Multiple Complementary Experts Under Heavy Class Imbalance}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2024},
  doi = {10.1109/jstars.2024.3369876},
}
Figure