Abstract
In remote sensing, semantic segmentation models often struggle with heavy class imbalance, where common "head" classes dominate rare "tail" classes. This paper introduces the Multiple Complementary Experts (MCE) framework, which utilizes specialized neural network heads to improve the detection of rare land cover types. By training experts on overlapping class subsets and employing a complementary loss function, the MCE framework achieves significant gains in tail class accuracy with minimal impact on overall performance.
Introduction
Semantic segmentation in remote sensing is frequently hindered by the "long-tail" distribution of land cover classes. In mountainous regions, for example, vast areas of forest and bedrock (head classes) often overshadow critical but rare features like glaciers or specific vegetation types (tail classes). Standard deep learning models, acting as "lazy" optimizers, tend to maximize overall accuracy by ignoring these infrequent classes. While various techniques exist for image-level classification imbalance, applying them to pixel-level semantic segmentation is challenging because multiple classes often coexist within a single image.
The Multiple Complementary Experts (MCE) Framework
The authors propose a Multiple Complementary Experts (MCE) architecture to address this imbalance. Rather than a single monolithic network, the framework employs a shared visual backbone that extracts general features, which are then processed by several specialized "expert" heads:
- Expert 1 (Generalist): Trained across all available classes.
- Expert 2 (Specialist): Focuses on "body" and "tail" classes (medium to rare frequency).
- Expert 3 (Niche Specialist): Focuses exclusively on the rarest "tail" classes.
To ensure these experts specialize effectively, two key mechanisms are introduced:
- Complementary Loss ($L_{com}$): This loss function penalizes experts for predicting classes outside their intended specialization, forcing them to "stay in their lane."
- Adaptive Learning Rates: Tail specialists receive lower learning rates to prevent overfitting and the "hallucination" of rare classes in inappropriate contexts.
Datasets and Evaluation
The framework was evaluated on two primary datasets:
- TLM Dataset: A new, high-resolution alpine dataset covering 2,300 $km^2$ of the Swiss Alps. It contains 10 classes with an extreme distribution where the most common class is 1,000 times more frequent than the rarest.
- FLAIR Dataset: A large-scale French dataset used to validate the model's generalizability across different geographic regions.
Results and the Aggregation Bottleneck
The MCE approach demonstrated substantial improvements over standard Cross-Entropy baselines. On the TLM dataset using a DeepLabv3+ backbone, accuracy for the rarest classes increased from 23.6% to 62.0%. This massive boost in tail performance came at a minor cost to Overall Accuracy (OA), which only decreased from 89.2% to 87.2%.
A critical finding of the study involves the aggregation module, which combines the expert opinions (logits) into a final prediction. The researchers compared several strategies, including simple averaging and a theoretical "Oracle" model.
The "Oracle" study—which assumes the model always knows which expert to trust for a given pixel—yielded significantly higher results than automated methods. This suggests that while training specialized experts is effective, the current frontier in this research lies in developing more sophisticated, learnable aggregation methods to better synthesize expert wisdom.
Conclusion
The MCE framework provides a robust and relatively simple solution for class imbalance in semantic segmentation. By allowing experts to have overlapping territories, the model fosters complementarity rather than competition. Future work will focus on moving beyond simple logit averaging toward context-aware aggregation sub-networks to further bridge the gap toward Oracle-level performance.
Summary updated