Parameter Efficient Self-Supervised Geospatial Domain Adaptation

Linus Scheibenreif, Michael Mommert, Damian BorthCVPR 2024 · DOI

TL;DR

This paper introduces Scaled Low-Rank (SLR) adapters, a parameter-efficient method to adapt large geospatial foundation models to new sensor modalities (like SAR or multi-spectral data) using unlabeled imagery. By training only 1-2% of the model’s parameters, the authors achieve a 4-6% boost in accuracy, often outperforming full-model fine-tuning while drastically reducing computational costs.

What Problem Does This Paper Address?

In the world of remote sensing, "data" isn't just one thing. While standard computer vision models thrive on RGB images, satellite sensors produce a "heterogeneous" mix: Synthetic Aperture Radar (SAR), multi-spectral bands (like near-infrared), and hyperspectral data.

Large-scale foundation models (like ViTs trained with Masked Autoencoding) are typically pre-trained on massive optical RGB datasets. When you try to use these models for a specific task—say, identifying crop types in SAR imagery—you hit two significant hurdles:

  1. Domain Shift: The model doesn't "understand" the statistical distribution of SAR data compared to RGB.
  2. Efficiency Gap: Fine-tuning a 300M+ parameter model is computationally expensive and requires a large amount of labeled data, which is rare and expensive to produce in the geospatial domain.

Methods

The authors propose a multi-stage adaptation framework designed to bridge the gap between general-purpose optical pre-training and specialized sensor tasks.

The proposed multi-stage framework for geospatial domain adaptation, including foundation model training (0), model adaptation (1), self-supervised domain adaptation (2), and supervised task adaptation (3).

As illustrated in the framework above, the process involves:

  1. Self-Supervised Domain Adaptation: Instead of jumping straight to a labeled task, the model is "primed" on unlabeled data from the target domain (e.g., raw SAR images). They use a Masked Autoencoding (MAE) objective, where the model tries to reconstruct missing patches of an image.
  2. Scaled Low-Rank (SLR) Adapters: This is the core technical novelty. Rather than updating the entire model, small "adapter" modules are inserted into the Transformer blocks (specifically the MSA and MLP layers).
  3. Mechanism: SLR freezes the original weights ($W_i$) and introduces two trainable scaling vectors ($s^1, s^2$) and two low-rank matrices ($W^1, W^2$).
  4. Scaling: Unlike standard Low-Rank Adaptation (LoRA), SLR uses element-wise scaling to adjust activations before and after the low-rank transformation. This allows the model to better handle the drastic shifts in pixel values found in non-optical sensors.

Datasets

The researchers evaluated their method across 8 diverse remote sensing datasets, covering various modalities and resolutions:

  • Optical/RGB: RESISC45, UCMerced, FireRisk.
  • Multi-spectral: EuroSAT, TreeSatAI, BENGE-8k (Sentinel-2).
  • SAR (Radar): EuroSAT-SAR, BENGE-8k (Sentinel-1).

These datasets vary in Ground Sampling Distance (GSD) from 0.2m to 30m, providing a rigorous test of the model's flexibility.

Key Insights and Ideas

  • Unlabeled Data is a Goldmine: Self-supervised training on unlabeled target data is more effective for domain adaptation than supervised training on small labeled sets. It allows the model to recover modality-specific details (like the grainy texture of SAR) before it ever sees a label.
  • Less is More: By training only ~1% of parameters, the model is less prone to "catastrophic forgetting" of the general features learned during large-scale pre-training. This is why SLR often beats full fine-tuning, especially when labeled data is scarce.
  • Scaling Matters for Sensors: Standard adapters often struggle with the high dynamic range of satellite data. The "scaling" vectors in SLR act as a learnable normalization layer that helps bridge the gap between RGB-trained weights and raw sensor values.

Key Results

The experimental results demonstrate that SLR adapters provide a superior balance between accuracy and computational efficiency compared to standard fine-tuning or linear evaluation.

Comparison of average accuracy and number of trainable parameters across different foundation models (MAE, SatMAE, Scale-MAE). SLR FT achieves higher accuracy than full fine-tuning while updating only ~1% of the parameters.
  • Accuracy Boost: SLR adapters improved downstream linear evaluation accuracy by 4-6% (absolute) across all 8 datasets. As shown in the chart above, SLR FT (green) consistently outperforms Full FT (red) while using a fraction of the parameters (gray bars).
  • Efficiency: The method outperforms full-model fine-tuning while updating only 1-2% of the parameters (roughly 3.9M to 7.1M parameters for a 303M parameter ViT-L).
  • Few-Shot Champion: In scenarios with very few labels ($k=10$ samples per class), SLR improved accuracy by 6-10% compared to standard fine-tuning. On the EuroSAT-SAR dataset, it reached >98% of the accuracy of a fully fine-tuned model with a fraction of the cost.
  • Memory Savings: Because the base model remains frozen, researchers can store one large foundation model and multiple tiny "adapter" files for different sensors, drastically reducing storage and memory footprints.

Limitations and Future Work

While SLR works for different visual modalities, the authors note it might struggle if the domain gap is too large (e.g., adapting a vision model to audio data). Furthermore, the current work focuses on static imagery; future research could extend SLR adapters to temporal data (satellite video or time-series) to capture seasonal changes. Finally, as with all remote sensing technology, the dual-use nature of the work requires responsible management regarding surveillance and privacy.

Summary updated

BibTeX
@inproceedings{scheibenreif2024parameter,
  author = {Linus Scheibenreif and Michael Mommert and Damian Borth},
  title = {{Parameter Efficient Self-Supervised Geospatial Domain Adaptation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02630},
}
Figure