Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation

Zhixiang Chi, Yanan Wu, Li Gu, Huan Liu, Ziqiang Wang, Yang Zhang, Yang Wang, Konstantinos N. Plataniotis

TL;DR

The paper introduces Feedback-driven Self-adaptive Attention (FSA), a training-free plug-in module that significantly improves CLIP's open-vocabulary semantic segmentation. By routing final semantic output predictions back into intermediate attention layers, FSA corrects localization errors and ensures spatial coherence without altering model parameters. It delivers substantial performance gains across multiple architectures and benchmarks with minimal computational overhead.

What Problem Does This Paper Address?

Vision-Language Models like CLIP are exceptionally good at matching whole images with text descriptions, but they struggle with dense, pixel-level tasks like semantic segmentation. Because CLIP is trained primarily on global image-text pairs, its internal patch-level features suffer from poor localization and spatial misalignment.

Existing training-free methods try to fix this by modifying CLIP's intermediate attention maps during the forward pass (for example, replacing standard attention with self-self patch relationships). However, the authors identify a major flaw in this approach: subsequent network operations (like projections and Feed-Forward Networks) inject noise and disrupt this carefully engineered spatial coherence before it ever reaches the final output. Furthermore, intermediate attention layers lack direct interaction with text representations.

Figure 1. Comparison with the existing training-free methods. Top: Prior works refine intermediate attention but face limitations: (1) improvements may not propagate to final segmentation; (2) at- tention lacks direct class information. Bottom: We introduce a feedback-driven self-adaptive mechanism that reintegrates seman- tic outputs with visual and textual cues into CLIP. Our approach is orthogonal to existing methods and serves as a plug-in.

Methods

FSA solves this by creating a feedback loop that uses the model's final output predictions—which contain the richest combination of visual and textual class information—to refine its intermediate attention maps. The framework consists of three main stages:

  1. Stand-alone Intermediate Attention Isolation: To ensure the feedback signal purely reflects the impact of the attention mechanism (and not background context), the authors run a parallel branch using a manually crafted uniform attention map. By subtracting the uniform branch's logits from the original branch's logits, they isolate the net semantic effect of the learned attention. Pairwise patch similarities are then calculated using KL divergence.
  2. Confidence-Based Sparse Attention: Not all patch relationships are useful; low-correlation patches can dilute the signal. FSA filters these out by sorting patch similarities and keeping only the most confident relationships up to a cumulative probability threshold ($p$). An adaptive exponential scaling factor ($\lambda$) is then applied to amplify the strongest connections.
  3. Feedback Self-Adaptation & Ensemble: The resulting sparse feedback attention map ($\mathcal{A}^f$) is reintegrated into the final layer of the CLIP vision encoder. The authors propose three distinct mathematical strategies to modulate how the attention interacts with the Value ($\boldsymbol {V}$) matrix and combine them into a single robust ensemble.
Figure 4. Overview of our feedback-driven self-adaptive attention framework. We isolate the contribution of initial attention using a dual branch with uniform attention. Pairwise patch similarity is then calculated by applying KL divergence to the output logits. A confidence-based pruning step follows to eliminate irrelevant patches and amplify the correlation between related patches. The resulting sparse feedback attention is adapted back to the initial attention maps using three operations, which are subsequently ensembled.

Datasets

The authors evaluated FSA across eight standard open-vocabulary segmentation benchmarks:

  • With background class: PASCAL VOC, PASCAL Context, and COCO Object.
  • Without background class: PASCAL VOC20, PASCAL Context59, COCO Stuff, Cityscapes, and ADE20K.

Testing spanned three Vision Transformer backbones from CLIP and OpenCLIP: ViT-B/16, ViT-L/14, and ViT-H/14.

Key Insights and Ideas

  • The Output as a Blueprint: Instead of guessing how to fix intermediate layers during the forward pass, the final output predictions should be treated as the most reliable semantic prior available, because they have already interacted with the text prompt embeddings.
  • The Residual Noise Trap: The paper empirically demonstrates that spatial coherence drops sharply after residual connections and FFN blocks in standard CLIP. Feedback mechanisms act as a corrective shield against this degradation.
  • Context Subtraction: Raw output logits contain a global "spread out" context. Introducing a uniform attention baseline allows the system to isolate exactly what the selective attention layer contributed, leading to a much cleaner feedback signal.

Key Results

FSA acts as a seamless plug-in, consistently boosting the performance of four state-of-the-art training-free segmentation methods (MaskCLIP, SCLIP, ClearCLIP, and ProxyCLIP):

  • Massive Boosts for Baseline Methods: When plugged into MaskCLIP, FSA improved mean Intersection over Union (mIoU) by +7.9 on ViT-B, +18.7 on ViT-L, and +14.1 on ViT-H.
  • Versatility Across Attention Types: FSA successfully enhanced standard Q-K attention, self-self attention, and Proxy attention configurations (augmented with external models like MAE, SAM, and DINO).
  • Negligible Overhead: Because it only modifies the final attention block, FSA adds a mere 3% to 5% computational overhead, making it highly efficient for deployment.

Qualitatively, FSA eliminates the "holes" and unstable edges common in zero-shot segmentation, forcing the model to group semantically similar parts of an object together cleanly.

Figure 6. Qualitative results. By integrating our feedback self- adaptive mechanism, we correct missegmented patches, ensuring consistent segmentation within each object.

Limitations and Future Work

Intuitively, a feedback loop could be run iteratively (feeding the adjusted outputs back in multiple times) to progressively refine the segmentation. However, the authors note that iterative adaptation did not yield further improvements in their experiments. Figuring out why the performance gains saturate after a single feedback step remains an open question for future research.

Summary updated

Abstract

CLIP exhibits strong visual-textual alignment but struggle with open-vocabulary segmentation due to poor localization. Prior methods enhance spatial coherence by modifying intermediate attention. But, this coherence isn't consistently propagated to the final output due to subsequent operations such as projections. Additionally, intermediate attention lacks direct interaction with text representations, such semantic discrepancy limits the full potential of CLIP. In this work, we propose a training-free, feedback-driven self-adaptive framework that adapts output-based patch-level correspondences back to the intermediate attention. The output predictions, being the culmination of the model's processing, encapsulate the most comprehensive visual and textual semantics about each patch. Our approach enhances semantic consistency between internal representations and final predictions by leveraging the model's outputs as a stronger spatial coherence prior. We design key modules, including attention isolation, confidence-based pruning for sparse adaptation, and adaptation ensemble, to effectively feedback the output coherence cues. Our method functions as a plug-in module, seamlessly integrating into four state-of-the-art approaches with three backbones (ViT-B, ViT-L, ViT-H). We further validate our framework across multiple attention types (Q-K, self-self, and Proxy augmented with MAE, SAM, and DINO). Our approach consistently improves their performance across eight benchmarks.
BibTeX
@inproceedings{chi2025plug,
  author = {Zhixiang Chi and Yanan Wu and Li Gu and Huan Liu and Ziqiang Wang and Yang Zhang and Yang Wang and Konstantinos N. Plataniotis},
  title = {{Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure