Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning

Neha Kalibhat, Priyatham Kattakinda, Sumit Nawathe, Arman Zarei, Nikita Seleznev, Samuel Sharpe, Senthil Kumar, Soheil Feizi

This paper, recently published by researchers from the University of Maryland and CapitalOne Research, challenges one of the fundamental building blocks of modern computer vision: the image patch. While Vision Transformers (ViTs) have revolutionized the field by treating images as a grid of small, uniform squares, this research suggests that we might be missing the forest for the (very pixelated) trees.

TL;DR

The authors propose replacing arbitrary grid-based image patches with "semantically meaningful tokens"—representing specific objects and their relationships—to improve visual representation learning. By explicitly encoding objects (tangible tokens) and their interactions (intangible tokens), their model achieves a staggering 47% improvement in text-to-image retrieval and significantly better compositional reasoning than standard Vision Transformers.

What Problem Does This Paper Address?

Standard Vision Transformers (ViTs) "patchify" an image into a grid (e.g., 16x16 squares). As the authors illustrate in Figure 2, these individual patches often lack independent meaning—a single patch might just be a blurry segment of "blue" that could be sky, water, or a marble.

Figure 2. Processing of text vs image data: In this simple illustration, we demonstrate the notable difference in how text and visual data are processed by humans and transformers. Humans are capable of deciphering larger concepts from images (both tangible and intangible), where each concept has independent semantic meaning.

Humans don't perceive the world as a grid; we see entities and the relationships between them. The authors argue that the grid-patch approach limits a model's ability to learn compositional representations—understanding not just that a "dog" and a "rock" exist, but specifically that the "dog is sitting on the rock."

Methods

The researchers developed a framework to extract and encode high-level semantic information before it ever hits the Transformer.

  1. Token Extraction: They use off-the-shelf models to decompose an image:
  2. Tangible Tokens: Instance segmentation masks (via the SEEM model) representing physical objects like "Dog," "Cat," or "Tree."
    • Intangible Tokens: Relationships and actions (via the RAM model) like "Beside" or "Sitting," which are embedded using a CLIP text encoder.
  3. Visual Token Encoder: Instead of raw pixels, a Transformer encoder processes these semantic tokens along with global image features.
  4. Additive Attention: To capture the structure of the scene, the authors introduce a novel attention mechanism. They rank relationships (e.g., subject-predicate-object triplets) and spatial proximity (K-nearest neighbors) to create a weight matrix that biases the Transformer's self-attention scores. This effectively tells the model, "pay more attention to the relationship between these two specific objects."
  5. Contrastive Alignment: The resulting image embeddings are aligned with text embeddings using a contrastive loss, similar to the training regime used for CLIP.
Figure 1. Training with meaningful visual tokens: We present a framework that uses off-the-shelf segmentation and relation extraction models to prepare a set of tangible tokens (V) and intangible tokens (U) for any arbitrary image, along with directional semantic relation- ships between them. These tokens and image features (l) are then passed as input to our visual token encoder (f(.)). We utilize the semantic relation (E) and relative location (N) information of all tokens to compute additive attention weights, ranked by importance. The learned image embeddings (s) are contrastively aligned with the text embeddings (t) of the CLIP text encoder (g(.)), which is simultaneously fine-tuned with our model.

Datasets

  • Training: The model was trained on the COCO (Common Objects in Context) dataset.
  • Evaluation: Performance was measured using standard retrieval tasks on COCO and two specialized compositionality benchmarks:
  • ARO (Attribution, Relation, and Order): Tests if models can distinguish between "a dog on a rug" and "a rug on a dog."
    • Winoground: A challenging benchmark where two images and two captions share the same words but in different semantic configurations.

Key Insights and Ideas

  • Semantic Grounding: By using object masks as tokens, the model starts with a "vocabulary" of visual entities that have inherent meaning, mirroring how language models process words.
  • The "Intangible" Advantage: Explicitly including relationship tokens (the "verbs" of the image) helps the model bridge the gap between visual data and natural language descriptions.
  • Structural Priors: The additive attention mechanism acts as a powerful inductive bias. By injecting scene-graph information directly into the attention layers, the model learns complex spatial and semantic hierarchies much faster than a standard ViT.

Key Results

The results demonstrate that semantic tokenization is a powerful alternative to grid-patching:

  • Retrieval Performance: Compared to a standard ViT-s/16 trained on the same data, the proposed model showed a +47% improvement in text-to-image retrieval and a +44% improvement in image-to-text retrieval.
  • Compositional Reasoning: On the ARO benchmark, the model outperformed ViT by 18%. On Winoground, it showed a 10% improvement.
  • Beating the Giants: Remarkably, the model (trained only on COCO) outperformed the original CLIP model (pre-trained on 400M images) on several compositional metrics, highlighting the efficiency of semantic tokenization.

Limitations and Future Work

While promising, the authors acknowledge several hurdles:

  • Preprocessing Overhead: The method relies on running heavy segmentation and relation extraction models for every image before training can begin. This adds significant computational cost and disk space requirements.
  • Dependency on Upstream Models: The quality of the visual representation is strictly capped by the accuracy of the off-the-shelf segmenters and relation extractors used.
  • Scaling: The current study is a "proof-of-concept" on COCO. Future work is needed to see if these benefits hold when scaling to massive, web-scale datasets.
  • End-to-End Integration: A major goal for future research is to develop a unified model that can perform scene-graph generation and representation learning simultaneously, rather than relying on separate pre-processing steps.

Summary updated

Abstract

Vision transformers have established a precedent of patchifying images into uniformly-sized chunks before processing. We hypothesize that this design choice may limit models in learning comprehensive and compositional representations from visual data. This paper explores the notion of providing semantically-meaningful visual tokens to transformer encoders within a vision-language pre-training framework. Leveraging off-the-shelf segmentation and scene-graph models, we extract representations of instance segmentation masks (referred to as tangible tokens) and relationships and actions (referred to as intangible tokens). Subsequently, we pre-train a vision-side transformer by incorporating these newly extracted tokens and aligning the resultant embeddings with caption embeddings from a text-side encoder. To capture the structural and semantic relationships among visual tokens, we introduce additive attention weights, which are used to compute self-attention scores. Our experiments on COCO demonstrate notable improvements over ViTs in learned representation quality across text-to-image (+47%) and image-to-text retrieval (+44%) tasks. Furthermore, we showcase the advantages on compositionality benchmarks such as ARO (+18%) and Winoground (+10%).
BibTeX
@inproceedings{kalibhat2025understanding,
  author = {Neha Kalibhat and Priyatham Kattakinda and Sumit Nawathe and Arman Zarei and Nikita Seleznev and Samuel Sharpe and Senthil Kumar and Soheil Feizi},
  title = {{Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning}},
  booktitle = {Workshops @ CVPR},
  year = {2025},
  url = {https://openreview.net/forum?id=CfHsFYr7rZ},
}
Figure