Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning

Neha Mukund Kalibhat, Priyatham Kattakinda, Arman Zarei, Nikita Seleznev, Samuel Sharpe, Senthil Kumar, Soheil Feizi

TL;DR

This paper proposes a novel tokenization strategy for vision transformers, moving beyond fixed-size image patches to semantically meaningful "tangible" (object) and "intangible" (relationship) tokens. This approach significantly improves visual representation learning, leading to state-of-the-art performance in text-to-image and image-to-text retrieval, as well as enhanced compositional reasoning.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have revolutionized computer vision by treating images as sequences of fixed-size patches, similar to how text is tokenized into words. However, this uniform patching approach might limit the model's ability to learn comprehensive and compositional representations. Unlike words, which inherently carry semantic meaning, image patches often lack independent meaning and require context from surrounding patches to be understood. This can hinder the model's understanding of complex scenes, especially when it comes to relationships between objects or abstract concepts. The paper aims to address this by exploring a tokenization strategy that imbues visual tokens with richer semantic information, akin to how humans process language.

Methods

The core innovation lies in a new tokenization method that leverages off-the-shelf models to extract semantically meaningful visual tokens.

  • Tangible Tokens: These represent visible objects in an image. The authors use an instance segmentation model (like SEEM) to extract mask embeddings for detected objects. These embeddings are hypothesized to capture richer object-specific information than standard patches.
  • Intangible Tokens: These represent relationships and actions between objects (e.g., "sitting on," "in front of"). These are extracted using a relation extraction model (like RAM) that analyzes pairs of object masks. These tokens capture semantic meaning that isn't directly visible as a discrete object.
  • Additive Attention Weights: To further enhance the model's understanding of the relationships between these new tokens, the authors introduce an "additive attention" mechanism. This mechanism assigns weights to self-attention scores based on the importance of semantic relationships (from intangible tokens and triplets) and relative spatial positions (from nearest neighbor information). These weights are ranked by importance, simulating how humans prioritize information.
  • Visual Token Encoder: A transformer model is trained using these enriched tokens, along with global image features. This encoder's output is then aligned with text embeddings from a pre-trained CLIP text encoder using a contrastive loss.

Datasets

  • COCO: Used for both training the segmentation model (SEEM) and for pre-training the Visual Token Encoder and fine-tuning the CLIP text encoder.
  • Visual Genome: Used in conjunction with COCO for training the relation extraction model (RAM).
  • ARO (Attribute, Relation, and Object) Benchmark: Used for evaluating compositional reasoning capabilities. It includes sub-datasets like VG-Relation, VG-Attribution, COCO-Order, and Flickr-Order.
  • Winoground Benchmark: Also used for evaluating compositional reasoning, specifically testing a model's ability to associate correct images with correct captions and vice versa.

Key Insights and Ideas

  • Beyond Patches: The fundamental idea is that fixed-size patches are a suboptimal way to represent visual information for transformers, especially when compared to how humans process images by identifying objects and their relationships.
  • Tangible + Intangible = Richer Understanding: Combining visible object representations (tangible tokens) with abstract relationship representations (intangible tokens) provides a more comprehensive understanding of an image's content and context.
  • Structured Attention: Explicitly incorporating relational and positional information through additive attention weights allows the model to better reason about the structure and semantics of visual scenes.
  • Efficiency Gains: While token extraction adds an initial overhead, the subsequent transformer processing on lower-dimensional, semantically rich tokens can be more efficient than processing raw image patches.

Key Results

  • Retrieval Performance: The proposed Visual Token Encoder (VTE) significantly outperforms standard ViT and even fine-tuned CLIP on text-to-image retrieval (+47% over ViT, +9% over CLIP) and image-to-text retrieval (+44% over ViT, +4% over CLIP).
  • Compositionality: The VTE shows substantial improvements on compositionality benchmarks: +18% on ARO and +10% on Winoground, demonstrating better understanding of relationships and attributes.
  • Additive Attention Benefit: The additive attention mechanism further boosts performance, particularly on tasks requiring nuanced understanding of relationships.

Limitations and Future Work

  • Scalability: The authors acknowledge that the performance of this tokenization approach in truly large-scale setups (beyond COCO) is an open question.
  • Unified Model: They suggest developing a unified model that can handle both scene graph generation and representation learning more seamlessly.
  • Efficiency: Making the tokenization process itself more compute and memory efficient is a key area for future research.
  • Alternative Tokenization: The paper notes that their token extraction method can be substituted with other segmentation or scene graph generation models, implying flexibility in the approach.

Summary updated

Abstract

Vision transformers have established a precedent of patchifying images into uniformly-sized chunks before processing. We hypothesize that this design choice may limit models in learning comprehensive and compositional representations from visual data. This paper explores the notion of providing semantically-meaningful visual tokens to transformer encoders within a vision-language pre-training framework. Leveraging off-the-shelf segmentation and scene-graph models, we extract representations of instance segmentation masks (referred to as tangible tokens) and relationships and actions (referred to as intangible tokens). Subsequently, we pre-train a vision-side transformer by incorporating these newly extracted tokens and aligning the resultant embeddings with caption embeddings from a text-side encoder. To capture the structural and semantic relationships among visual tokens, we introduce additive attention weights, which are used to compute self-attention scores. Our experiments on COCO demonstrate notable improvements over ViTs in learned representation quality across text-to-image (+47%) and image-to-text retrieval (+44%) tasks. Furthermore, we showcase the advantages on compositionality benchmarks such as ARO (+18%) and Winoground (+10%).
BibTeX
@inproceedings{kalibhat2025understanding,
  author = {Neha Mukund Kalibhat and Priyatham Kattakinda and Arman Zarei and Nikita Seleznev and Samuel Sharpe and Senthil Kumar and Soheil Feizi},
  title = {{Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning}},
  booktitle = {Workshops @ CVPR},
  year = {2025},
  url = {https://openreview.net/forum?id=CfHsFYr7rZ},
}
Figure