TL;DR
FrogDogNet is a novel prompt learning framework that significantly improves the domain generalization of CLIP for remote sensing scene classification. By integrating a Fourier Filter Block to isolate robust low-frequency structural features and introducing a remote sensing-specific prompt alignment loss, it effectively discards background noise and adapts to unseen domain shifts.
What Problem Does This Paper Address?
Large-scale Vision-Language Models (VLMs) like CLIP have revolutionized computer vision through zero-shot inference. However, adapting them to Remote Sensing (RS) tasks—such as satellite or aerial image classification—remains a massive challenge.
RS images suffer from severe domain shifts caused by variations in sensor modalities, atmospheric conditions, seasons, and geographic locations. Furthermore, crucial objects of interest (like buildings or airplanes) are often small and scattered against massive, homogeneous backgrounds (like oceans or deserts). Existing prompt learning methods (e.g., CoCoOp) condition text prompts on full-image features indiscriminately. In remote sensing, this causes the model to overfit to highly variable background clutter and high-frequency noise, leading to poor generalization on unseen target domains.
Methods
The authors introduce FrogDogNet (Fourier frequency Retained visual prompt Output Guidance), a framework designed to filter out irrelevant background noise and retain only the most discriminative, class-invariant features for prompt learning.
The architecture consists of three core components:
- Projection Network & Self-Attention Module: Refines the initial image embeddings from CLIP's visual encoder to capture complex spatial and contextual dependencies.
- Fourier Filter Block (FFB): This block converts the processed image features from the spatial domain to the frequency domain using the Fast Fourier Transform (FFT). It retains only the top $k$ low-frequency components (LFCs)—which capture stable semantic structures—and discards high-frequency components associated with noise. An Inverse FFT (IFFT) maps these filtered features back to the spatial domain.
- Remote Sensing Prompt Alignment (RPA) Loss: To ensure the learned prompts generalize well, the authors introduce a novel loss function ($L_{RPA}$). It minimizes the Euclidean distance between the model's learned prompts and CLIP's original text embeddings across multiple RS-specific text templates (e.g., "a photo of a [CLS]", "satellite photo of a [CLS]").
Datasets
The framework was evaluated on four widely used remote sensing benchmarks:
- PatternNet: 38 classes, 800 images per class.
- RSICD: 30 classes, 10,000 images total.
- RESISC45: 45 classes, 700 images per class.
- MLRSNet: 46 classes, 109,161 images total.
For specialized domain generalization tasks, the authors utilized the v2 versions of these datasets, filtering them down to 16 identical overlapping classes to strictly evaluate performance across domain shifts.
Key Insights and Ideas
- The Frequency Domain Advantage: In remote sensing, class-defining structures (like the layout of a harbor or airport) reside primarily in the low-frequency spectrum. High-frequency spectrums typically contain sensor noise or transient environmental details. Filtering in the frequency domain provides a cleaner abstraction than spatial cropping.
- The $k$-Frequency Sweet Spot: Completely removing high frequencies can blur critical object boundaries. The authors discovered a sensitivity threshold: retaining 350 out of 512 low-frequency components yields the optimal balance between noise reduction and structural preservation.
- Prompt Harmonization: Forcing the learned visual prompts to align with diverse textual descriptions (via the RPA loss) prevents the text encoder from drifting during few-shot training, preserving CLIP's foundational zero-shot capabilities.
Key Results
FrogDogNet was tested across three challenging generalization paradigms: Base-to-New (B2N) class generalization, Cross-Dataset (CD) transfer, and Single-Source Multi-Target Domain Generalization (DG).
- Base-to-New Generalization: FrogDogNet achieved the highest Harmonic Mean (HM) scores across all datasets, outperforming the previous state-of-the-art method, StyLIP, by an average of 3.91%.
- Cross-Dataset Transfer: When trained on PatternNet and tested on completely unseen datasets, FrogDogNet outperformed StyLIP by 6.98% on RSICD, 6.41% on RESISC45, and 5.36% on MLRSNet.
- Feature Space Separation: t-SNE visualizations demonstrate that FrogDogNet forms tightly packed, well-separated class clusters compared to CoCoOp, which suffers from significant class overlap.
- Computational Efficiency: Despite the addition of FFT operations, FrogDogNet maintains a highly efficient profile at 192.361 GFLOPS, making it faster than competing methods like APPLeNet.
- Backbone Robustness: The authors verified performance across multiple vision backbones (RN50, RN101, ViT-B16, ViT-B32), with ViT-B16 consistently yielding the highest accuracy.
Limitations and Future Work
While FrogDogNet demonstrates exceptional performance, the paper notes a few areas for improvement:
- Fixed Frequency Threshold: The number of retained low-frequency components ($k=350$) is statically set based on empirical sensitivity analysis. An adaptive mechanism that dynamically adjusts $k$ based on image characteristics could be more robust.
- Scope of Tasks: The evaluation is strictly limited to scene classification. Future work is needed to explore how frequency-based visual prompting translates to dense pixel-level prediction tasks like remote sensing object detection and semantic segmentation.
Summary updated