Semantic Consistency of Vision Tokens: A Vision-Centric Perspective on Multimodal Large Language Models

Nikola Đukić, Tim Lebailly, Tinne TuytelaarsarXiv 2026 · arXiv

TL;DR

Multimodal Large Language Models (MLLMs) often rely on vision encoders that sacrifice local detail for global language alignment, leading to poor performance on "dense" tasks like counting or spatial reasoning. This paper introduces Semantically-Guided Training (SGT), a novel objective that forces vision tokens to be semantically consistent with local image content while maintaining their "readability" by the language model, setting a new state-of-the-art across a wide range of vision-centric benchmarks.

What Problem Does This Paper Address?

Current MLLMs (like LLaVA or GPT-4o) typically use vision encoders like CLIP or SigLIP. These models are "language-aligned," meaning they are trained to match global image features with text descriptions. While this makes them excellent at understanding high-level concepts, it often comes at the cost of patch-level quality.

In these encoders, individual image patches often lose their local semantic meaning. Some patches even become "semantically inconsistent," acting as global information sinks rather than representing the specific objects they cover. This is a major hurdle for "vision-centric" tasks that require fine-grained spatial understanding—think of a robot trying to count specific items on a shelf or an AI explaining the relative depth of objects in a complex scene.

Previous attempts to fix this by simply swapping CLIP for a high-quality self-supervised model (like DINOv2) or fusing the two have largely failed because the resulting features no longer "speak the same language" as the LLM.

Methods

The authors propose a two-pronged approach to bridge this gap:

  1. Formalizing Semantic (In)consistency: They define a metric called Semantic Inconsistency (SIC). It measures how much a vision token's similarity to other tokens in the same image deviates from a "gold standard" self-supervised model (DINOv2). This allows them to categorize tokens into Semantically Consistent Tokens (SCTs), which represent local content well, and Semantically Inconsistent Tokens (SITs), which capture global noise.
  2. Semantically-Guided Training (SGT): Instead of just using the standard next-token prediction loss during MLLM instruction tuning, they add a second "semantic guidance" term. This new loss explicitly minimizes the SIC of the vision tokens.

Crucially, the vision encoder is unfrozen during this process. The joint objective pushes the encoder to produce tokens that are both semantically grounded (thanks to the guidance loss) and language-aligned (thanks to the LLM's feedback).

Figure 1: Top panel: Analyzed image (first column), Principal Components Analysis (PCA) visu- alizations of features produced by DINOv2 and CLIP (second and third columns) and semantic inconsistency (eq. 1) for each CLIP token. Bottom panel: cosine similarities between the highlighted patch and all other image patches, where ✓denotes semantically consistent tokens (SCTs) and × denotes semantically inconsistent tokens (SITs).

Datasets

The models were trained using the standard LLaVA 1.5 instruction tuning recipe. Evaluation was performed across a diverse suite of benchmarks:

  • Text-centric: TextVQA (OCR), SciQA (science questions), GQA (general reasoning), and POPE (hallucination).
  • Vision-centric: TallyQA (counting), RefCOCO/OCID-Ref (localization via text), VSR (spatial reasoning), and CV-Bench (a generalist vision-centric benchmark covering depth, distance, and counting).

Key Insights and Ideas

  • MLLMs are SCT-hungry: Through a "probing" experiment, the authors found that removing just a small percentage of semantically consistent tokens (SCTs) causes a much sharper drop in performance than removing inconsistent ones (SITs). This proves that MLLMs want to use consistent local information but are often forced to sift through noise.
Figure 2: MLLM probing on CV-Bench by removing top-k (shown on x-axis) semantically consistent tokens (SCTs, full line) or semantically inconsistent tokens (SITs, dotted line). Left: LLaVA 1.5 model family. Right: Gemma 3 model family. Increments between points correspond to roughly 1% of the total number of vision tokens (6 for LLaVA and 3 for Gemma).
  • SITs are more than just "Registers": Previous work identified a few "register tokens" in ViTs that capture global info. This paper shows that SITs are far more numerous and widespread than just a few high-norm tokens, making them a systemic issue for dense vision tasks.
  • Joint Training is the "Secret Sauce": Training a vision backbone to be more local before plugging it into an MLLM (like Locality-Aligned CLIP) helps, but it’s not optimal. SGT performs better because the vision encoder learns exactly which semantic features the LLM needs to solve the task at hand.

Key Results

SGT achieves a new state-of-the-art average score across all nine evaluated benchmarks.

  • Vision-Centric Gains: On the CV-Bench, SGT improved the LLaVA 1.5 7B baseline from 55.21% to 57.85%. On RefCOCO (localization), it jumped from 77.54% to 83.17%.
  • Maintaining Textual Prowess: Unlike many methods that boost vision at the expense of language, SGT retains strong performance on text-centric tasks like GQA and SciQA.
  • Generalization: The method proved effective when scaling from 7B to 13B parameters and worked across different vision backbones (CLIP vs. SigLIP) and LLM architectures (Vicuna vs. Qwen).

Limitations and Future Work

The primary limitation acknowledged is computational cost. Because SGT requires a full forward and backward pass through the LLM to update the vision encoder, it is currently more suitable for fine-tuning/instruction tuning rather than massive-scale pretraining from scratch.

Future work could explore more efficient ways to distill this semantic consistency into the vision encoder or investigate if the "gold standard" reference (DINOv2) can be replaced by a more lightweight or learned guidance signal.

Summary updated

Abstract

In computer vision, vision-language and self-supervised encoders are two unique paradigms characterized by global language alignment and high-quality vision-only patch representations, respectively. Multimodal large language models (MLLMs) often use the former because of language alignment, despite the recent rise in vision-centric tasks resembling traditional dense vision tasks that would likely benefit from the high-quality patch representations, raising the question whether self-supervised models could aid MLLMs in vision-centric tasks. To investigate this, we define semantic (in)consistency of patch tokens from language-aligned encoders and identify two sets of tokens, semantically consistent (SCTs) and semantically inconsistent (SITs). Through systematic probing via the removal of SCTs or SITs from the input of modern MLLMs, we show higher reliance on SCTs than SITs, as removal of the former induces larger performance drops. This demonstrates that MLLMs benefit from semantic consistency of vision tokens. With this in mind, we devise a simple yet effective extension of the MLLM training objective that combines traditional language modeling with semantic guidance via minimization of semantic inconsistency. This strategy, named Semantically-Guided Training (SGT), combines the signals of both loss terms to push the vision encoder towards providing semantically consistent tokens, while preserving language alignment. Extensive evaluation shows that SGT outperforms all baselines and MLLMs with specialized vision encoders on vision-centric benchmarks while keeping strong scores on text-centric benchmarks, thus setting state of the art on average across all tasks. Our results highlight the importance of combining language alignment and semantic consistency through the lens of MLLMs rather than independently from them and pave the way towards the design of better MLLM vision encoders.
BibTeX
@misc{ukic2026semantic,
  author = {Nikola Đukić and Tim Lebailly and Tinne Tuytelaars},
  title = {{Semantic Consistency of Vision Tokens: A Vision-Centric Perspective on Multimodal Large Language Models}},
  year = {2026},
  eprint = {2611.15448},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15448},
}
Figure