Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP

Eunji Kim, Kyuhong Shim, Simyung Chang, Sungroh Yoon

Abstract

Vision-Language Models (VLMs) like CLIP have transformed cross-modal tasks, yet their text encoders remain largely opaque "black boxes." We present SToRI (Semantic Token Reweighting for Interpretable text embeddings), a framework that introduces fine-grained control and interpretability to CLIP. By modifying the self-attention mechanism to weight individual tokens, SToRI allows for both data-driven discovery of discriminative features and user-driven modulation of retrieval preferences. Our results demonstrate that SToRI achieves state-of-the-art performance in few-shot classification while providing human-interpretable insights into model decision-making.

Introduction

While CLIP effectively bridges images and text, its standard text encoder generates a single, monolithic embedding where the specific contribution of each word is obscured. This leads to two primary challenges: a lack of interpretability regarding which words drive classification, and a lack of controllability for users who wish to emphasize specific attributes in a prompt (e.g., focusing on "blonde hair" over "eyeglasses"). SToRI addresses these gaps by allowing users and algorithms to "turn up the volume" on specific semantic tokens within a prompt.

Methodology

The core of SToRI is Semantic Token Reweighting (STR). Rather than altering the model architecture or adding external adapters, STR modifies the internal self-attention mechanism of the Transformer. By introducing a weight ($w_n$) for each token $n$, the attention scores are recalculated to selectively amplify or dampen the influence of specific words during the encoding process.

The SToRI framework architecture

As shown in the framework overview, SToRI supports two primary modes of operation:

  1. Data-driven Control: Weights are learned via backpropagation on specific datasets, revealing which semantic elements are most discriminative for a given category.
  2. User-driven Control: Humans manually adjust weights to reflect personal search intent, enabling precise, preference-based image retrieval.

Empirical testing determined that applying this reweighting from the 7th block of the CLIP text encoder onward provides the optimal balance between flexibility and embedding stability.

Interpretability and Insights

A key advantage of SToRI is its ability to align model behavior with human intuition. In classification tasks, the model automatically learns to prioritize semantically meaningful words. For instance, when trained on the Describable Textures Dataset (DTD), SToRI assigns high weights to words like "multiple" and "stripes" for the 'Banded' category, and "spider web" for the 'Cobwebbed' category.

Visualization of learned token weights for texture classification

Furthermore, SToRI captures context-dependent importance. In fine-grained bird classification (CUB dataset), the word "striped" may receive a high weight only when it serves as a primary differentiator between two similar species. This suggests that SToRI effectively modulates the meaningful components of human language rather than relying on non-semantic statistical noise.

Results and Performance

SToRI was evaluated across diverse benchmarks, including ImageNet, SUN397, and CelebA.

  • Few-Shot Classification: SToRI outperforms state-of-the-art methods like TaskRes. In 1-shot learning on ImageNet, SToRI achieved 76.68% accuracy (vs. TaskRes's 74.69%) while maintaining full interpretability.
  • Controllable Retrieval: In preference-based retrieval tests on the CelebA dataset, increasing an attribute's weight (e.g., from 1.0 to 1.5) significantly boosted Average Precision (AP), allowing users to find specific visual features faster.
  • Efficiency: The framework is highly practical, adding only a 1.02x runtime overhead compared to standard CLIP.

Conclusion

SToRI successfully bridges the gap between high-performance VLMs and the need for human-centric control and transparency. While the system's efficacy is currently bounded by the quality of the initial text descriptions, it offers a robust path forward for making large-scale models more interpretable and responsive to user intent. Future work will explore extending STR to multi-modal tasks and even larger vision-language architectures.

Summary updated

Abstract

A text encoder within Vision-Language Models (VLMs) like CLIP plays a crucial role in translating textual input into an embedding space shared with images, thereby facilitating the interpretative analysis of vision tasks through natural language. Despite the varying significance of different textual elements within a sentence depending on the context, efforts to account for variation of importance in constructing text embeddings have been lacking. We propose a framework of Semantic Token Reweighting to build Interpretable text embeddings (SToRI), which incorporates controllability as well. SToRI refines the text encoding process in CLIP by differentially weighting semantic elements based on contextual importance, enabling finer control over emphasis responsive to data-driven insights and user preferences. The efficacy of SToRI is demonstrated through comprehensive experiments on few-shot image classification and image retrieval tailored to user preferences.
BibTeX
@inproceedings{kim2024semantic,
  author = {Eunji Kim and Kyuhong Shim and Simyung Chang and Sungroh Yoon},
  title = {{Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.837},
}
Figure