Introduction
High-resolution satellite imagery is generated at an unprecedented scale, yet the manual annotation required for semantic segmentation—labeling every pixel as a building, road, or tree—remains a significant bottleneck. This paper, presented at WACV 2023, addresses the "label gap" in remote sensing by proposing a self-supervised framework that learns robust representations without extensive human labeling.
The Challenge: Labels and Domain Shift
Semantic segmentation in aerial imagery faces two primary hurdles:
- The Label Gap: While raw data is abundant, high-quality pixel-level labels are scarce. Supervised models trained on limited datasets often fail to generalize.
- Domain Shift: Models trained on one geographic region (e.g., Potsdam) frequently underperform when applied to another (e.g., Nice) due to variations in architecture, vegetation, and sensor characteristics.
Traditional contrastive learning (CL) methods often focus on global image features, which are insufficient for the dense, pixel-level requirements of segmentation.
Proposed Method: mCL-LC
The authors introduce mCL-LC (multi-level Contrastive Learning with Local Consistency), a framework designed to capture both high-level semantics and fine-grained local details.
Multi-level Contrastive Learning
Unlike standard approaches that apply CL only at the encoder level, mCL-LC introduces a second contrastive loss at the decoder level. This ensures that the semantic outputs remain consistent across different image augmentations, forcing the model to learn task-specific features suitable for dense prediction.
Local Consistency (LC) via Mutual Information
To preserve sharp boundaries and fine details, the framework employs Mutual Information (MI) maximization. By selecting matching regions of interest (ROIs) across augmented views and maximizing their MI, the model maintains local semantic smoothness and structural integrity.
Pseudo-Cloud Augmentation
Weather conditions, particularly cloud cover, are a major source of noise in satellite data. The authors developed a "pseudo-cloud" generator to simulate cloud clusters during training. This forces the model to learn "cloud-invariant" features, effectively teaching it to ignore atmospheric interference.
Experimental Results
The model was evaluated on the ISPRS Potsdam, Vaihingen, and MiniFrance datasets. The latter is particularly challenging, involving 16 land-cover categories and significant geographical shifts between training and testing cities.
As shown in the figure above, mCL-LC (f) produces segmentation maps that are significantly closer to the ground truth (b) than competing methods like DeepLab v3+ (c) or GLCNet (e). The proposed method handles the complex textures and boundaries of the aerial scene with much higher fidelity.
Key Performance Metrics
- Accuracy: On the Nice dataset, mCL-LC achieved an Overall Accuracy (OA) of 69.44%, outperforming GLCNet (64.94%). With pseudo-cloud augmentation (mCL-LC+), accuracy reached 71.20%.
- Zero-Shot Generalization: In a cross-city test (training on Nice, testing on Nantes Saint), mCL-LC outperformed the next best competitor by over 15%, demonstrating superior robustness to domain shift.
Conclusion and Future Work
mCL-LC represents a significant step forward in self-supervised remote sensing. By combining multi-level contrastive learning with local consistency constraints and domain-specific augmentations, the framework achieves state-of-the-art performance and remarkable generalization. Future research may explore extending this approach to multi-modal data (e.g., combining RGB with LiDAR) and reducing the computational overhead associated with Mutual Information estimation.
Summary updated