C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote Sensing

Avigyan Bhattacharya, Mainak Singha, Ankit Jha, Biplab Banerjee

This paper, titled "C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote Sensing," presents a new framework for improving how large-scale vision-language models (like CLIP) handle the unique challenges of satellite and aerial imagery.

TL;DR

C-SAW is a novel framework that enhances CLIP for remote sensing by combining self-supervised jigsaw reconstruction with "visual attentive" prompt learning. It significantly improves the model's ability to generalize across different geographic domains and unseen object classes, even when dealing with distorted or scrambled image data.

What Problem Does This Paper Address?

Remote sensing (RS) imagery is notoriously difficult for standard computer vision models. Unlike everyday photos, RS data suffers from massive "domain shifts"—the same type of forest or city looks different depending on the sensor, the altitude, the weather, or the geographic region.

While Vision-Language Models (VLMs) like CLIP are powerful, they have two major weaknesses in this field:

  1. Spatial Blindness: CLIP’s vision encoder often treats images as a "bag of features," struggling to understand the contextual relationship between different parts of an image. This is a problem in RS, where the spatial arrangement of land-cover patches is vital.
  2. Static Prompts: Existing methods to "tune" CLIP often use generic or static text prompts that don't account for the specific "style" or domain of a satellite image.

The authors specifically highlight that models like APPLeNet (a previous state-of-the-art) see a performance drop when image patches are jumbled—a common occurrence due to data transmission errors in satellite communications.

Methods

The authors introduce C-SAW, which keeps the CLIP backbone frozen but adds several clever components:

  • Self-Supervised Jigsaw Task: The model is trained to take a "jumbled" image (where patches are shuffled) and reconstruct the original. This forces the vision encoder to learn the spatial context and structural "logic" of RS scenes.
  • Visual Attentive Text Prompting (VATP): Instead of using a fixed prompt like "a photo of a forest," C-SAW generates "Visual Attentive Tokens." It extracts style statistics (mean features) from intermediate layers of the vision encoder and uses them to condition the text prompts. This makes the prompts dynamic and domain-aware.
  • Multi-Objective Loss: The model is optimized using a combination of four losses:
  • Cross-entropy ($L_{ce}$): For standard classification.
  • Self-supervised ($L_{ssl}$): Using a Barlow Twins-style approach to align features of jumbled and original images.
  • Reconstruction ($L_{recon}$): To ensure the model can physically "un-jumble" the image.
  • Diversity Maximization ($L_{dm}$): To ensure the learned prompts are distinct and expressive.

Datasets

The researchers evaluated C-SAW on five major remote sensing benchmarks:

  • PatternNet: 38 classes, high-resolution.
  • RSICD: 30 classes, focused on scene description.
  • RESISC45: 45 classes, a standard for scene classification.
  • MLRSNet: A large-scale multi-label dataset (46 classes).
  • EuroSat: 10 classes based on Sentinel-2 satellite data.

They also used "v2" versions of these datasets to test Single Source Multi-Target (SSMT) generalization, where a model trained on one dataset is tested on all others.

Key Insights and Ideas

  • Spatial Context is a Superpower: By forcing the model to solve a "jigsaw puzzle" of satellite patches, C-SAW learns a much more robust representation of land cover than models that just look at global statistics.
  • Style as a Signal: The "aha moment" here is using intermediate layer statistics as a proxy for "domain style." By feeding this style back into the text encoder, the model effectively "translates" the visual domain into the text space.
  • Robustness to Corruption: C-SAW is significantly more resilient to scrambled data. While other models' accuracy plummets when patches are out of order, C-SAW maintains high performance because it was trained to understand those spatial relationships.

Key Results

C-SAW consistently outperformed state-of-the-art (SOTA) methods like MaPLe and APPLeNet across three main tasks:

  • Base-to-Novel Class Generalization: C-SAW improved the Harmonic Mean (a balance of seen and unseen class accuracy) by 2-6% across all datasets.
  • Cross-Dataset Generalization: When trained on PatternNet and tested on others, C-SAW beat zero-shot CLIP by 27.2% on the source domain and 4.46% on average across target domains.
  • SSMT Generalization: It achieved a 4.94% average improvement over target domains compared to previous SOTA methods.
  • Jumbled Images: As shown in Figure 1, C-SAW maintains an accuracy of 80.32% on jumbled inputs where APPLeNet scores 78.43%.

Limitations and Future Work

The authors note that while C-SAW is excellent at generalizing to new domains, it doesn't yet have a dedicated mechanism for outlier identification (detecting images that don't belong to any known class). They suggest that future work could integrate anomaly detection to make the model even more reliable for real-world autonomous monitoring. Additionally, the computational overhead of the reconstruction block during training is a factor, though the backbone remains frozen.

Summary updated

Abstract

We focus on domain and class generalization problems in analyzing optical remote sensing images, using the large-scale pre-trained vision-language model (VLM), CLIP. While contrastively trained VLMs show impressive zero-shot generalization performance, their effectiveness is limited when dealing with diverse domains during training and testing. Existing prompt learning techniques overlook the importance of incorporating domain and content information into the prompts, which results in a drop in performance while dealing with such multi-domain data. To address these challenges, we propose a solution that ensures domain-invariant prompt learning while enhancing the expressiveness of visual features. We observe that CLIP's vision encoder struggles to identify contextual image information, particularly when image patches are jumbled up. This issue is especially severe in optical remote sensing images, where land-cover classes exhibit well-defined contextual appearances. To this end, we introduce C-SAW, a method that complements CLIP with a self-supervised loss in the visual space and a novel prompt learning technique that emphasizes both visual domain and content-specific features. We keep the CLIP backbone frozen and introduce a small set of projectors for both the CLIP encoders to train C-SAW contrastively. Experimental results demonstrate the superiority of C-SAW across multiple remote sensing benchmarks and different generalization tasks.
BibTeX
@inproceedings{bhattacharya2023c,
  author = {Avigyan Bhattacharya and Mainak Singha and Ankit Jha and Biplab Banerjee},
  title = {{C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote Sensing}},
  booktitle = {ICVGIP},
  year = {2023},
  doi = {10.1145/3627631.3627669},
}

Loading related content...

Figure