FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

Haoyang Li, Liang Wang, Siyu Zhou, Jiacheng Sun, Jing Jiang, Chao Wang, Guodong Long, Yan Peng

TL;DR

FVG-PT (Foreground View-Guided Prompt Tuning) is a plug-and-play enhancement for Vision-Language Models (VLMs) like CLIP that fixes a common but overlooked failure mode: the tendency for internal attention to drift away from the main object toward the background during fine-tuning. By adaptively guiding the model to focus on the foreground while balancing it with original model priors, FVG-PT significantly improves both task-specific accuracy and generalization to unseen classes across multiple architectures.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) like CLIP have revolutionized computer vision, but adapting them to specific downstream tasks usually requires "prompt tuning"—adding a few learnable parameters while keeping the massive foundation model frozen.

The authors identify a critical issue: as these prompts are optimized, the model’s internal attention distribution shifts. In many failure cases, the model stops "looking" at the foreground object (e.g., a car) and starts focusing on irrelevant background features (e.g., the road or sky). This "attention shift" prevents the model from properly aligning visual semantics with text. Furthermore, existing methods that try to force foreground focus often suffer from the Base-New Trade-off (BNT): they become so specialized on the training classes (base) that they lose the general "commonsense" knowledge needed to recognize unseen classes (new).

Methods

FVG-PT introduces three modular components to create an adaptive, foreground-aware tuning process:

  1. Foreground Reliability Gate (FRG): The system first uses an external segmentation model (SEEM) to generate a foreground mask. Because these masks can be noisy or incorrect, the FRG uses a small Multi-Layer Perceptron (MLP) to calculate a "trust score" ($r$). It evaluates the mask based on prediction entropy, similarity to the full image, and geometric area.
  2. Foreground Distillation Compensation (FDC): This is the core "guidance" module. It inserts lightweight adapters into the visual and textual branches. Using the trust score from the FRG, it applies a distillation loss that encourages the model's full-image representation to align with the features of the isolated foreground.
  3. Prior Calibration (PC): To solve the BNT problem, the authors decouple the inference paths. For unseen classes, the PC module uses a "Backbone Reliability Gate" to adaptively balance the tuned model's output with the original, frozen CLIP prior. This ensures the model doesn't "over-focus" on foreground features at the expense of general knowledge.

Datasets

The authors evaluated FVG-PT on 11 diverse datasets commonly used in the VLM community:

  • General Objects: ImageNet, Caltech101.
  • Fine-grained Categories: OxfordPets, Food101, StanfordCars, Flowers102, FGVCAircraft.
  • Specialized Domains: EuroSAT (satellite imagery), DTD (textures), SUN397 (scenes), and UCF101 (action recognition).

Key Insights and Ideas

  • Attention as the Culprit: The paper provides empirical evidence (via Grad-CAM visualizations) that prompt tuning failures are directly linked to the visual encoder's attention deviating from the foreground.
  • Adaptive Trust is Essential: Unlike previous methods that treat foreground masks as ground truth, FVG-PT acknowledges that masks can be "incomplete or overly large." The learnable Reliability Gate allows the model to ignore the mask if it's unhelpful.
  • Decoupling for Generalization: One of the most elegant ideas is the structural decoupling of the "base" and "new" branches. By isolating the optimization paths, the model can be aggressive in learning foreground features for known tasks without destroying its zero-shot capabilities for unknown ones.
  • Data Efficiency: FVG-PT is remarkably efficient. The results show that FVG-PT using only 50% of the training data can match the performance of standard prompt tuning (CoOp) using 100% of the data.

Key Results

The authors tested FVG-PT as a "plug-in" for four major backbone models: CoOp, KgCoOp, PromptSRC, and MMRL.

  • Performance Boost: FVG-PT consistently improved the Harmonic Mean (HM) of base and new class accuracies across all backbones. For example, it boosted the baseline CoOp model's average HM from 75.09% to 77.76%.
  • Cross-Dataset Transfer: When trained on ImageNet and tested on the other 10 datasets, FVG-PT showed superior transferability, proving that foreground-guided features are more robust across different data distributions.
  • Low Overhead: Despite the added complexity, FVG-PT only adds about 0.13M trainable parameters (for a ViT-B/16 backbone), making it highly parameter-efficient.

Limitations and Future Work

  • Architecture Dependency: FVG-PT is designed for VLMs with both visual and textual branches. It is not directly applicable to "visual-only" prompt tuning methods like VPT that lack a text encoder branch for alignment.
  • Preprocessing Cost: While the authors note that the segmentation model (SEEM) runs at a high FPS (~190), obtaining foreground masks still adds a preprocessing step to the pipeline.
  • Future Directions: The authors suggest exploring self-supervised variants that could identify foreground regions without needing an external pretrained segmentation model, which would further simplify the deployment of FVG-PT.

Summary updated

Abstract

CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT. Codes are available at: https://github.com/JREion/FVG-PT
BibTeX
@misc{li2026fvg,
  author = {Haoyang Li and Liang Wang and Siyu Zhou and Jiacheng Sun and Jing Jiang and Chao Wang and Guodong Long and Yan Peng},
  title = {{FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models}},
  year = {2026},
  eprint = {2603.08708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.08708},
}
Figure