FrogDogNet: Fourier frequency Retained visual prompt Output Guidance for Domain Generalization of CLIP in Remote Sensing

Hariseetharam Gunduboina, Muhammad Haris Khan, Biplab Banerjee

TL;DR

FrogDogNet is a novel prompt learning framework that significantly improves the domain generalization of CLIP for remote sensing scene classification. By integrating a Fourier Filter Block to isolate robust low-frequency structural features and introducing a remote sensing-specific prompt alignment loss, it effectively discards background noise and adapts to unseen domain shifts.

What Problem Does This Paper Address?

Large-scale Vision-Language Models (VLMs) like CLIP have revolutionized computer vision through zero-shot inference. However, adapting them to Remote Sensing (RS) tasks—such as satellite or aerial image classification—remains a massive challenge.

RS images suffer from severe domain shifts caused by variations in sensor modalities, atmospheric conditions, seasons, and geographic locations. Furthermore, crucial objects of interest (like buildings or airplanes) are often small and scattered against massive, homogeneous backgrounds (like oceans or deserts). Existing prompt learning methods (e.g., CoCoOp) condition text prompts on full-image features indiscriminately. In remote sensing, this causes the model to overfit to highly variable background clutter and high-frequency noise, leading to poor generalization on unseen target domains.

Methods

The authors introduce FrogDogNet (Fourier frequency Retained visual prompt Output Guidance), a framework designed to filter out irrelevant background noise and retain only the most discriminative, class-invariant features for prompt learning.

Overview of the FrogDogNet architecture, combining a projection network, self-attention, and the Fourier Filter Block (FFB) to guide prompt generation.

The architecture consists of three core components:

  1. Projection Network & Self-Attention Module: Refines the initial image embeddings from CLIP's visual encoder to capture complex spatial and contextual dependencies.
  2. Fourier Filter Block (FFB): This block converts the processed image features from the spatial domain to the frequency domain using the Fast Fourier Transform (FFT). It retains only the top $k$ low-frequency components (LFCs)—which capture stable semantic structures—and discards high-frequency components associated with noise. An Inverse FFT (IFFT) maps these filtered features back to the spatial domain.
  3. Remote Sensing Prompt Alignment (RPA) Loss: To ensure the learned prompts generalize well, the authors introduce a novel loss function ($L_{RPA}$). It minimizes the Euclidean distance between the model's learned prompts and CLIP's original text embeddings across multiple RS-specific text templates (e.g., "a photo of a [CLS]", "satellite photo of a [CLS]").
Figure 3. The Fourier Filter Block (FFB) transforms image em- beddings X using Fast Fourier Transform (FFT), retains the top k frequencies, and applies Inverse FFT (IFFT) to map them back to the original embedding space.

Datasets

The framework was evaluated on four widely used remote sensing benchmarks:

  • PatternNet: 38 classes, 800 images per class.
  • RSICD: 30 classes, 10,000 images total.
  • RESISC45: 45 classes, 700 images per class.
  • MLRSNet: 46 classes, 109,161 images total.

For specialized domain generalization tasks, the authors utilized the v2 versions of these datasets, filtering them down to 16 identical overlapping classes to strictly evaluate performance across domain shifts.

Key Insights and Ideas

  • The Frequency Domain Advantage: In remote sensing, class-defining structures (like the layout of a harbor or airport) reside primarily in the low-frequency spectrum. High-frequency spectrums typically contain sensor noise or transient environmental details. Filtering in the frequency domain provides a cleaner abstraction than spatial cropping.
  • The $k$-Frequency Sweet Spot: Completely removing high frequencies can blur critical object boundaries. The authors discovered a sensitivity threshold: retaining 350 out of 512 low-frequency components yields the optimal balance between noise reduction and structural preservation.
Figure 1. Impact of Fourier Filtering on RS Image Analysis: (a) demonstrates that retaining 50% of low-frequency components (LFCs) preserves structural details while reducing noise in the fre- quency magnitude spectrum. (b) presents a sensitivity analysis, showing that keeping 350 out of 512 LFCs of visual features fv(x) achieves the highest average generalization performance.
  • Prompt Harmonization: Forcing the learned visual prompts to align with diverse textual descriptions (via the RPA loss) prevents the text encoder from drifting during few-shot training, preserving CLIP's foundational zero-shot capabilities.

Key Results

FrogDogNet was tested across three challenging generalization paradigms: Base-to-New (B2N) class generalization, Cross-Dataset (CD) transfer, and Single-Source Multi-Target Domain Generalization (DG).

  • Base-to-New Generalization: FrogDogNet achieved the highest Harmonic Mean (HM) scores across all datasets, outperforming the previous state-of-the-art method, StyLIP, by an average of 3.91%.
  • Cross-Dataset Transfer: When trained on PatternNet and tested on completely unseen datasets, FrogDogNet outperformed StyLIP by 6.98% on RSICD, 6.41% on RESISC45, and 5.36% on MLRSNet.
  • Feature Space Separation: t-SNE visualizations demonstrate that FrogDogNet forms tightly packed, well-separated class clusters compared to CoCoOp, which suffers from significant class overlap.
Figure 4. t-SNE plots [103] depicting the image features extracted from the Meta-Net of CoCoOp and the FFB of FrogDogNet for the domain generalization (DG) task on the RSICDv2 dataset. The legends indicate the corresponding class labels.
  • Computational Efficiency: Despite the addition of FFT operations, FrogDogNet maintains a highly efficient profile at 192.361 GFLOPS, making it faster than competing methods like APPLeNet.
  • Backbone Robustness: The authors verified performance across multiple vision backbones (RN50, RN101, ViT-B16, ViT-B32), with ViT-B16 consistently yielding the highest accuracy.
Figure 8. Variation in FrogDogNet’s classification performance with different backbone models for the B2N class generalization task across all datasets. The harmonic mean (HM) of base and new class accuracies is used as the evaluation metric.

Limitations and Future Work

While FrogDogNet demonstrates exceptional performance, the paper notes a few areas for improvement:

  • Fixed Frequency Threshold: The number of retained low-frequency components ($k=350$) is statically set based on empirical sensitivity analysis. An adaptive mechanism that dynamically adjusts $k$ based on image characteristics could be more robust.
  • Scope of Tasks: The evaluation is strictly limited to scene classification. Future work is needed to explore how frequency-based visual prompting translates to dense pixel-level prediction tasks like remote sensing object detection and semantic segmentation.

Summary updated

Abstract

In recent years, large-scale vision-language models (VLMs) like CLIP have gained attention for their zero-shot inference using instructional text prompts. While these models excel in general computer vision, their potential for domain generalization in remote sensing (RS) remains underexplored. Existing approaches enhance prompt learning by generating visual prompt tokens but rely on full-image features, introducing noise and background artifacts that vary within a class, causing misclassification. To address this, we propose FrogDogNet, a novel prompt learning framework integrating Fourier frequency filtering and self-attention to improve RS scene classification and domain generalization. FrogDogNet selectively retains invariant low-frequency components while eliminating noise and irrelevant backgrounds, ensuring robust feature representation across domains. The model first extracts significant features via projection and self-attention, then applies frequency-based filtering to preserve essential structural information for prompt learning. Extensive experiments on four RS datasets and three domain generalization tasks show that FrogDogNet consistently outperforms state-of-the-art prompt learning methods, demonstrating superior adaptability across domain shifts. Our findings highlight the effectiveness of frequency-based invariant feature retention in generalization, paving the way for broader applications. Our code is available at https://github.com/HariseetharamG/FrogDogNet
BibTeX
@inproceedings{gunduboina2025frogdognet,
  author = {Hariseetharam Gunduboina and Muhammad Haris Khan and Biplab Banerjee},
  title = {{FrogDogNet: Fourier frequency Retained visual prompt Output Guidance for Domain Generalization of CLIP in Remote Sensing}},
  booktitle = {Workshops @ CVPR},
  year = {2025},
}
Figure