Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification

Long Lan, Fengxiang Wang, Xiangtao Zheng, Zengmao Wang, Xinwang Liu

TL;DR

Researchers have developed a novel prompt-tuning method that adapts large Vision-Language Models (like CLIP) for the tricky task of identifying specific ship types in satellite imagery. By integrating remote sensing "priors" and a hierarchical description system, their approach significantly improves the model's ability to recognize new ship categories it hasn't seen during training, while preventing it from getting distracted by complex maritime backgrounds.

What Problem Does This Paper Address?

Fine-Grained Ship Classification in Remote Sensing (RS-FGSC) is notoriously difficult. Unlike identifying a "dog" vs. a "cat," distinguishing a "destroyer" from a "frigate" in a low-resolution satellite image requires spotting very subtle structural differences.

Current methods face three major hurdles:

  1. Data Scarcity: Labeled data for rare ship classes is hard to find due to cost and privacy.
  2. The Domain Gap: Foundation models like CLIP are trained on natural "internet" images. When shown satellite photos, they often focus on the background (the ocean or docks) rather than the ship itself.
  3. Generalization: Models often overfit to the specific ships they saw during training (base classes) and fail miserably when asked to identify a new type of ship (new classes).

Methods

The authors propose a "Remote Sensing Prompt-Tuning" method that keeps the massive CLIP encoders frozen (saving computation) while adding lightweight, trainable components:

  • Remote Sensing Ship Priors: They use a specialized image encoder (pre-trained on the MillionAID remote sensing dataset) to extract features specific to satellite imagery. These features are fed into two small networks (Remote-Net and Visual-Net) that generate "bias terms." These biases act as a nudge, adjusting CLIP’s general features to be more "aware" of maritime contexts.
  • Hierarchical, Multi-Granularity Prompts: Instead of a simple prompt like "a photo of a [CLASS]," they use a structured taxonomy. A ship is described across three levels (e.g., Primary: Military vessel $\rightarrow$ Secondary: Destroyer $\rightarrow$ Final: Asagiri-class).
  • Learnable Context: They build on the CoOp architecture, using learnable vectors that adapt to the specific task, but they "condition" these vectors with the remote sensing bias to ensure the text and image features stay aligned.

Datasets

The study evaluates the method on several benchmarks:

  • FGSC-23 & FGSCR-42: Standard datasets for fine-grained ship classification.
  • EuroSAT: A land-use dataset used to test how well the method transfers to other remote sensing tasks.
  • FGSCM-52 (New): The authors introduced this expanded dataset, which adds 10 new categories and more samples for rare classes to FGSCR-42, providing a more robust testbed for "base-to-new" generalization.

Key Insights and Ideas

  • Background Suppression: One of the most striking insights is visual. Standard CLIP often "looks" at the water around a ship to make a decision. By adding RS-specific priors, the model's attention maps shift dramatically to focus on the actual hull and structure of the vessel.
  • Taxonomy Matters: Using a hierarchical prompt (Primary $\rightarrow$ Secondary $\rightarrow$ Final type) provides the model with a "roadmap" of features. This helps the model learn that certain features (like a flat deck) are shared across many "Military vessels," making it easier to generalize to a new military ship it hasn't seen before.
  • Efficiency: Because the core CLIP model remains frozen, the method is highly efficient to train, requiring only a small number of "shots" (labeled examples) to achieve high performance.

Key Results

The authors used the Harmonic Mean (H) to balance accuracy on "seen" (base) and "unseen" (new) classes:

  • Superior Generalization: In a 6-shot setting on the FGSCR-42 dataset, the proposed method achieved a Harmonic Mean of 32.55%, significantly outperforming the baseline CoOp (21.43%) and CoCoOp (21.74%).
  • Base Class Accuracy: On the FGSC-23 dataset (6-shot), the model reached 43.63% accuracy on base classes, nearly double the performance of some existing prompt-tuning methods.
  • Robustness: The method showed consistent improvements across different backbones (ViT-B/16 and ResNet-50) and even on non-ship datasets like EuroSAT, proving the "bias term" approach is a versatile way to bridge the domain gap.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Encoder Alignment: The RS-specific encoder and the CLIP encoder are currently treated as separate entities. Better alignment or joint training of these "priors" could yield even higher accuracy.
  • Resolution: Performance improves significantly at higher resolutions (e.g., 336x336 vs 224x224), suggesting that the model is still limited by the raw information available in smaller image patches.
  • Architectural Exploration: The current "bias" networks are simple two-layer bottlenecks; the authors suggest that more advanced network architectures for these adapters could be explored in the future.

Summary updated

Abstract

Remote-sensing fine-grained ship classification (RS-FGSC) poses a significant challenge due to the high similarity between classes and the limited availability of labeled data, limiting the effectiveness of traditional supervised classification methods. Recent advancements in large pretrained vision-language models (VLMs) have demonstrated impressive capabilities in few-shot or zero-shot learning, particularly in understanding image content. This study delves into harnessing the potential of VLMs to enhance classification accuracy for unseen ship categories, which holds considerable significance in scenarios with restricted data due to cost or privacy constraints. Directly fine-tuning VLMs for RS-FGSC often encounters the challenge of overfitting the seen classes, resulting in suboptimal generalization to unseen classes, which highlights the difficulty in differentiating complex backgrounds and capturing distinct ship features. To address these issues, we introduce a novel prompt tuning technique that employs a hierarchical, multigranularity prompt design. Our approach integrates remote sensing ship priors through bias terms, learned from a small trainable network. This strategy enhances the model’s generalization capabilities while improving its ability to discern intricate backgrounds and learn discriminative ship features. Furthermore, we contribute to the field by introducing a comprehensive dataset, FGSCM-52, significantly expanding existing datasets with more extensive data and detailed annotations for less common ship classes. Extensive experimental evaluations demonstrate the superiority of our proposed method over current state-of-the-art techniques. The source code will be made publicly available.
BibTeX
@article{lan2025efficient,
  author = {Long Lan and Fengxiang Wang and Xiangtao Zheng and Zengmao Wang and Xinwang Liu},
  title = {{Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2025},
  url = {https://openreview.net/forum?id=2bmRmWf76L},
}
Figure