C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote Sensing

Avigyan Bhattacharya, Mainak Singha, Ankit Jha, Biplab Banerjee

Abstract: We focus on domain and class generalization problems in analyzing optical remote sensing images, using the large-scale pre-trained vision-language model (VLM), CLIP. While contrastively trained VLMs show impressive zero-shot generalization performance, their effectiveness is limited when dealing with diverse domains during training and testing. Existing prompt learning techniques overlook the importance of incorporating domain and content information into the prompts, which results in a drop in performance while dealing with such multi-domain data. To address these challenges, we propose a solution that ensures domain-invariant prompt learning while enhancing the expressiveness of visual features. We observe that CLIP's vision encoder struggles to identify contextual image information, particularly when image patches are jumbled up. This issue is especially severe in optical remote sensing images, where land-cover classes exhibit well-defined contextual appearances. To this end, we introduce C-SAW, a method that complements CLIP with a self-supervised loss in the visual space and a novel prompt learning technique that emphasizes both visual domain and content-specific features. We keep the CLIP backbone frozen and introduce a small set of projectors for both the CLIP encoders to train C-SAW contrastively. Experimental results demonstrate the superiority of C-SAW across multiple remote sensing benchmarks and different generalization tasks.

Introduction

Remote sensing (RS) imagery presents unique challenges for standard computer vision models. Unlike everyday photography, RS data suffers from significant "domain shifts"—the same land-cover type can appear drastically different depending on the sensor, altitude, weather, or geographic region. While Vision-Language Models (VLMs) like CLIP are powerful, they exhibit two major weaknesses in this field: spatial blindness and static prompting. CLIP often treats images as a "bag of features," failing to grasp the vital spatial relationships between land-cover patches. Furthermore, standard prompt tuning methods use static text that ignores the specific "style" or domain of a satellite image.

The C-SAW Framework

To overcome these limitations, we introduce C-SAW, a framework that enhances CLIP while keeping its backbone frozen. C-SAW integrates two core components designed to improve spatial awareness and domain adaptation.

The C-SAW architecture, featuring the self-supervised jigsaw reconstruction task and Visual Attentive Text Prompting (VATP).

Self-Supervised Jigsaw Task

The model is trained on a "jumbled" image task where patches are shuffled. By forcing the model to reconstruct the original scene, the vision encoder learns the underlying structural logic and spatial context of RS imagery. This makes the model significantly more resilient to data corruption or transmission errors common in satellite communications.

Visual Attentive Text Prompting (VATP)

Instead of static prompts, C-SAW utilizes Visual Attentive Tokens. It extracts style statistics from the intermediate layers of the vision encoder to condition the text prompts. This makes the prompts dynamic and domain-aware, effectively translating visual style into the text space.

Experimental Results

C-SAW was evaluated against state-of-the-art (SOTA) methods like MaPLe and APPLeNet across five major benchmarks: PatternNet, RSICD, RESISC45, MLRSNet, and EuroSat. The framework demonstrated superior performance in three key areas:

  1. Base-to-Novel Class Generalization: C-SAW improved the Harmonic Mean (balancing seen and unseen class accuracy) by 2-6% across all datasets.
  2. Cross-Dataset Generalization: When trained on PatternNet and tested on others, C-SAW outperformed zero-shot CLIP by an average of 4.46% on target domains.
  3. Robustness to Corruption: As shown in the comparison below, C-SAW maintains high accuracy even when inputs are jumbled, a scenario where previous SOTA models see significant performance degradation.
Comparison of C-SAW and APPLeNet on puzzled (jumbled) inputs across three generalization tasks.

Conclusion

C-SAW effectively addresses the spatial and domain-specific limitations of CLIP for remote sensing. By combining self-supervised reconstruction with visual attentive prompting, it achieves new levels of generalization and robustness. Future work will focus on integrating anomaly detection to identify outliers, further enhancing the model's reliability for real-world autonomous monitoring.

Summary updated

Abstract

We focus on domain and class generalization problems in analyzing optical remote sensing images, using the large-scale pre-trained vision-language model (VLM), CLIP. While contrastively trained VLMs show impressive zero-shot generalization performance, their effectiveness is limited when dealing with diverse domains during training and testing. Existing prompt learning techniques overlook the importance of incorporating domain and content information into the prompts, which results in a drop in performance while dealing with such multi-domain data. To address these challenges, we propose a solution that ensures domain-invariant prompt learning while enhancing the expressiveness of visual features. We observe that CLIP's vision encoder struggles to identify contextual image information, particularly when image patches are jumbled up. This issue is especially severe in optical remote sensing images, where land-cover classes exhibit well-defined contextual appearances. To this end, we introduce C-SAW, a method that complements CLIP with a self-supervised loss in the visual space and a novel prompt learning technique that emphasizes both visual domain and content-specific features. We keep the CLIP backbone frozen and introduce a small set of projectors for both the CLIP encoders to train C-SAW contrastively. Experimental results demonstrate the superiority of C-SAW across multiple remote sensing benchmarks and different generalization tasks.
BibTeX
@inproceedings{bhattacharya2023c,
  author = {Avigyan Bhattacharya and Mainak Singha and Ankit Jha and Biplab Banerjee},
  title = {{C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote Sensing}},
  booktitle = {ICVGIP},
  year = {2023},
  doi = {10.1145/3627631.3627669},
}
Figure