TL;DR
This paper introduces a novel semi-supervised semantic segmentation framework for remote sensing images that enhances consistency regularization by applying perturbations not only at the image level but also at the feature level. By incorporating contrastive learning with an entropy-based threshold for feature sampling, the method achieves state-of-the-art performance on challenging remote sensing datasets, improving the accuracy and robustness of segmentation models.
What Problem Does This Paper Address?
Semantic segmentation of remote sensing images is crucial for applications like urban planning and environmental monitoring. However, manually annotating these images is extremely time-consuming and labor-intensive, making semi-supervised learning a necessity. Existing semi-supervised methods often struggle with the complex and densely distributed foreground categories in remote sensing imagery, leading to inconsistent predictions and challenges in feature space optimization. This paper aims to address these limitations by developing a more robust and accurate semi-supervised segmentation approach specifically tailored for the unique characteristics of remote sensing data.
Methods
The proposed framework builds upon the Mean Teacher (MT) architecture, a popular semi-supervised learning method. The key innovations are:
- Feature Perturbation: Unlike traditional MT that only perturbs input images, this method introduces perturbations directly to the feature maps extracted by the encoder. This is achieved through a "Feature Disturbed Module" (FDM) that applies both strong (e.g., Virtual Adversarial Training - VAT) and weak (e.g., feature jittering) perturbations to the features. This forces the model to learn more robust feature representations.
- Contrastive Learning with Entropy Threshold: To further optimize the feature space and ensure consistency after feature perturbation, the paper employs contrastive learning. A novel aspect is the use of an "Entropy Threshold Assisted Feature Sampling" strategy. By calculating the entropy of pixel-wise prediction probabilities, the method can identify uncertain or confused pixels. An entropy threshold is then used to select more reliable positive and negative keys for contrastive learning, thereby improving the precision of feature discrimination.
- Hybrid Loss Function: The overall loss function combines supervised loss (from labeled data), unsupervised consistency loss (from perturbed unlabeled data), and the proposed contrastive loss.
Datasets
The proposed method was evaluated on two widely used remote sensing datasets:
- iSAID: A large-scale and challenging benchmark for semantic segmentation in aerial imagery, featuring 15 object categories.
- Potsdam: A benchmark dataset derived from high-resolution aerial images of the Potsdam region in Germany, known for its detailed ground information and categories like buildings and streets.
The experiments utilized various labeled-to-unlabeled data ratios (1/8, 1/4, 1/2, and full dataset) to assess performance under different supervision levels.
Key Insights and Ideas
- Feature-level perturbations are crucial: Applying perturbations directly to feature maps, in addition to image-level augmentations, significantly enhances model robustness and generalization for complex remote sensing scenes.
- Entropy is a powerful tool for feature selection: Using pixel-wise prediction entropy to filter negative keys in contrastive learning is an effective way to handle the noisy and uncertain feature distributions common in remote sensing imagery, leading to more precise feature discrimination.
- Combining strong and weak perturbations is optimal: The study found that using both strong (VAT) and weak (feature jittering) perturbations for feature disturbance yielded the best results, suggesting a synergistic effect.
- Modular design contributes significantly: Ablation studies demonstrated that each proposed module (FDM, contrastive learning, entropy threshold) individually contributes to performance improvements, with their combination leading to substantial gains.
Key Results
The proposed method achieved state-of-the-art performance on both the iSAID and Potsdam datasets across various labeled data ratios.
- iSAID Dataset: The method achieved the best mIoU scores for 1/8, 1/4, and 1/2 labeled data proportions, with results of 42.65%, 45.08%, and 49.11%, respectively. It was also competitive with the top-performing methods on the full dataset, trailing the best by only 0.45% mIoU. Notably, it showed a distinct advantage in segmenting difficult categories like "Helicopter" and "Bridge."
- Potsdam Dataset: The method also achieved the best overall performance, with mIoU scores of 79.33%, 85.01%, and 85.93% for 1/8, 1/4, and 1/2 proportions, respectively. While the gap was smaller due to the distinct nature of Potsdam classes, it still outperformed other methods, particularly in categories like "Buildings," "Trees," and "Cars."
- Ablation Studies: Adding the FDM module alone increased mIoU by 2.32%. Adding contrastive learning increased mIoU by 1.11%. Incorporating the entropy threshold further improved mIoU by 0.69%. The full model achieved a total mIoU increase of 3.85% compared to the baseline.
Limitations and Future Work
- Static Entropy Threshold: The current method uses a static entropy threshold. The authors suggest that a dynamically adapting entropy threshold, which evolves with model training, could further optimize performance. This is identified as a promising direction for future research.
- Dataset Specificity: While the method performs well on iSAID and Potsdam, further validation on a wider variety of remote sensing datasets with diverse characteristics would be beneficial.
Summary updated