Semantic Segmentation in Aerial Imagery Using Multi-level Contrastive Learning with Local Consistency

Maofeng Tang, Konstantinos Georgiou, Hairong Qi, Cody Champion, Marc Bosch

Introduction

High-resolution satellite imagery is generated at an unprecedented scale, yet the manual annotation required for semantic segmentation—labeling every pixel as a building, road, or tree—remains a significant bottleneck. This paper, presented at WACV 2023, addresses the "label gap" in remote sensing by proposing a self-supervised framework that learns robust representations without extensive human labeling.

The Challenge: Labels and Domain Shift

Semantic segmentation in aerial imagery faces two primary hurdles:

  1. The Label Gap: While raw data is abundant, high-quality pixel-level labels are scarce. Supervised models trained on limited datasets often fail to generalize.
  2. Domain Shift: Models trained on one geographic region (e.g., Potsdam) frequently underperform when applied to another (e.g., Nice) due to variations in architecture, vegetation, and sensor characteristics.

Traditional contrastive learning (CL) methods often focus on global image features, which are insufficient for the dense, pixel-level requirements of segmentation.

Proposed Method: mCL-LC

The authors introduce mCL-LC (multi-level Contrastive Learning with Local Consistency), a framework designed to capture both high-level semantics and fine-grained local details.

Multi-level Contrastive Learning

Unlike standard approaches that apply CL only at the encoder level, mCL-LC introduces a second contrastive loss at the decoder level. This ensures that the semantic outputs remain consistent across different image augmentations, forcing the model to learn task-specific features suitable for dense prediction.

Local Consistency (LC) via Mutual Information

To preserve sharp boundaries and fine details, the framework employs Mutual Information (MI) maximization. By selecting matching regions of interest (ROIs) across augmented views and maximizing their MI, the model maintains local semantic smoothness and structural integrity.

Pseudo-Cloud Augmentation

Weather conditions, particularly cloud cover, are a major source of noise in satellite data. The authors developed a "pseudo-cloud" generator to simulate cloud clusters during training. This forces the model to learn "cloud-invariant" features, effectively teaching it to ignore atmospheric interference.

Experimental Results

The model was evaluated on the ISPRS Potsdam, Vaihingen, and MiniFrance datasets. The latter is particularly challenging, involving 16 land-cover categories and significant geographical shifts between training and testing cities.

Qualitative comparison of semantic segmentation results across different self-supervised methods.

As shown in the figure above, mCL-LC (f) produces segmentation maps that are significantly closer to the ground truth (b) than competing methods like DeepLab v3+ (c) or GLCNet (e). The proposed method handles the complex textures and boundaries of the aerial scene with much higher fidelity.

Key Performance Metrics

  • Accuracy: On the Nice dataset, mCL-LC achieved an Overall Accuracy (OA) of 69.44%, outperforming GLCNet (64.94%). With pseudo-cloud augmentation (mCL-LC+), accuracy reached 71.20%.
  • Zero-Shot Generalization: In a cross-city test (training on Nice, testing on Nantes Saint), mCL-LC outperformed the next best competitor by over 15%, demonstrating superior robustness to domain shift.

Conclusion and Future Work

mCL-LC represents a significant step forward in self-supervised remote sensing. By combining multi-level contrastive learning with local consistency constraints and domain-specific augmentations, the framework achieves state-of-the-art performance and remarkable generalization. Future research may explore extending this approach to multi-modal data (e.g., combining RGB with LiDAR) and reducing the computational overhead associated with Mutual Information estimation.

Summary updated

Abstract

Semantic segmentation in large-scale aerial images is an extremely challenging task. On one hand, the limited ground truth, as compared to the vast area the images cover, greatly hinders the development of supervised representation learning. On the other hand, the large footprint from remote sensing raises new challenges for semantic segmentation. In addition, the complex and ever changing image acquisition conditions further complicate the problem where domain shifting commonly occurs. In this paper, we exploit self-supervised contrastive learning (CL) methodologies for semantic segmentation in aerial imagery. In addition to performing CL at the feature level as most practices do, we add another level of contrastive learning, at the semantic level, taking advantage of the segmentation output from the downstream task. Further, we embed local mutual information in the semantic-level CL to enforce local consistency. This has largely enhanced the representation power at each pixel and improved the generalization capacity of the trained model. We refer to the proposed approach as multi-level contrastive learning with local consistency (mCL-LC). The experimental results on different benchmarks indicate that the proposed mCL-LC exhibits superior performance as compared to other state-of-the-art contrastive learning frameworks for the semantic segmentation task. mCL-LC also carries better generalization capacity especially when domain shifting exists.
BibTeX
@inproceedings{tang2023semantic,
  author = {Maofeng Tang and Konstantinos Georgiou and Hairong Qi and Cody Champion and Marc Bosch},
  title = {{Semantic Segmentation in Aerial Imagery Using Multi-level Contrastive Learning with Local Consistency}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00379},
}
Figure