Abstract
Vision-Language Models (VLMs) like CLIP often lack fine-grained controllability and interpretability due to their "black box" nature. We present SToRI (Semantic Token Reweighting for Interpretable text embeddings), a framework that modifies CLIP's internal attention mechanism to assign varying importance to individual text tokens. SToRI enables users to manually adjust word influence for precise image retrieval and allows for data-driven learning of token weights to explain classification decisions. Our results demonstrate that SToRI achieves state-of-the-art performance in few-shot classification while providing transparent, context-dependent explanations with minimal computational overhead.
Introduction
Standard CLIP models compress entire text prompts into a single embedding vector, treating all words with equal structural weight. This approach limits a user's ability to refine search results—for instance, emphasizing "yellow eyes" over "large size" when searching for specific owl species. Furthermore, it is often unclear which parts of a text description the model relies on for classification.
SToRI addresses these challenges by introducing Semantic Token Reweighting (STR). By "turning up the volume" on specific words, SToRI provides a mechanism for both human-in-the-loop control and algorithmic interpretability.
Methodology
The core of SToRI is the modification of the self-attention mechanism within CLIP’s transformer blocks. Instead of standard attention, SToRI injects a weight $w_n$ for each token $n$:
$$\hat{a}_{m,n} = \frac{w_n \exp(q_m k_n^T)}{\sum_j w_j \exp(q_m k_j^T)}$$
Increasing $w_n$ makes the $n$-th token more influential in the final embedding. The authors found that applying this reweighting from the 7th block onwards in the text encoder provides the best balance between semantic flexibility and model stability.
The framework supports two primary workflows:
- User-driven Control: Humans manually adjust weights to refine retrieval (e.g., emphasizing "blonde hair" while de-emphasizing "eyeglasses").
- Data-driven Control: Weights are learned automatically from a small set of images (few-shot learning), revealing which descriptors are most discriminative for a given category.
Results and Interpretability
SToRI was evaluated across diverse benchmarks, including ImageNet, CUB (birds), and the Describable Textures Dataset (DTD).
Few-shot Classification and Retrieval
SToRI performs competitively with or better than existing methods like TaskRes. In 1-shot learning on the DTD dataset, SToRI achieved an accuracy of 65.82%, demonstrating its ability to leverage CLIP's pre-trained knowledge effectively. In retrieval tasks on the CelebA dataset, users successfully shifted search priority by adjusting weights, significantly improving the Area Under the Curve (AUC) for targeted attributes.
Interpretability and Context
A key advantage of SToRI is that interpretability is "free." When weights are learned for a task, they serve as a natural explanation. For example, in texture classification, the model automatically prioritizes highly descriptive terms like "spider web" for the "cobwebbed" category.
Furthermore, SToRI captures context-dependent importance. The weight of a word like "striped" may change depending on which specific bird species are being compared, reflecting how human experts use different features for different diagnostic tasks. This efficiency comes with almost zero computational overhead, adding only about 1.02x to the runtime of standard CLIP.
Conclusion
SToRI provides a lightweight and effective method for making CLIP more controllable and transparent. While it remains dependent on the quality of the initial text prompt and the inherent biases of the base model, it represents a significant step toward more interactive and explainable Vision-Language Models. Future work may extend this token-level reweighting to spatial regions within images.
Summary updated