TL;DR
Multimodal Large Language Models (MLLMs) often rely on vision encoders that sacrifice local detail for global language alignment, leading to poor performance on "dense" tasks like counting or spatial reasoning. This paper introduces Semantically-Guided Training (SGT), a novel objective that forces vision tokens to be semantically consistent with local image content while maintaining their "readability" by the language model, setting a new state-of-the-art across a wide range of vision-centric benchmarks.
What Problem Does This Paper Address?
Current MLLMs (like LLaVA or GPT-4o) typically use vision encoders like CLIP or SigLIP. These models are "language-aligned," meaning they are trained to match global image features with text descriptions. While this makes them excellent at understanding high-level concepts, it often comes at the cost of patch-level quality.
In these encoders, individual image patches often lose their local semantic meaning. Some patches even become "semantically inconsistent," acting as global information sinks rather than representing the specific objects they cover. This is a major hurdle for "vision-centric" tasks that require fine-grained spatial understanding—think of a robot trying to count specific items on a shelf or an AI explaining the relative depth of objects in a complex scene.
Previous attempts to fix this by simply swapping CLIP for a high-quality self-supervised model (like DINOv2) or fusing the two have largely failed because the resulting features no longer "speak the same language" as the LLM.
Methods
The authors propose a two-pronged approach to bridge this gap:
- Formalizing Semantic (In)consistency: They define a metric called Semantic Inconsistency (SIC). It measures how much a vision token's similarity to other tokens in the same image deviates from a "gold standard" self-supervised model (DINOv2). This allows them to categorize tokens into Semantically Consistent Tokens (SCTs), which represent local content well, and Semantically Inconsistent Tokens (SITs), which capture global noise.
- Semantically-Guided Training (SGT): Instead of just using the standard next-token prediction loss during MLLM instruction tuning, they add a second "semantic guidance" term. This new loss explicitly minimizes the SIC of the vision tokens.
Crucially, the vision encoder is unfrozen during this process. The joint objective pushes the encoder to produce tokens that are both semantically grounded (thanks to the guidance loss) and language-aligned (thanks to the LLM's feedback).
Datasets
The models were trained using the standard LLaVA 1.5 instruction tuning recipe. Evaluation was performed across a diverse suite of benchmarks:
- Text-centric: TextVQA (OCR), SciQA (science questions), GQA (general reasoning), and POPE (hallucination).
- Vision-centric: TallyQA (counting), RefCOCO/OCID-Ref (localization via text), VSR (spatial reasoning), and CV-Bench (a generalist vision-centric benchmark covering depth, distance, and counting).
Key Insights and Ideas
- MLLMs are SCT-hungry: Through a "probing" experiment, the authors found that removing just a small percentage of semantically consistent tokens (SCTs) causes a much sharper drop in performance than removing inconsistent ones (SITs). This proves that MLLMs want to use consistent local information but are often forced to sift through noise.
- SITs are more than just "Registers": Previous work identified a few "register tokens" in ViTs that capture global info. This paper shows that SITs are far more numerous and widespread than just a few high-norm tokens, making them a systemic issue for dense vision tasks.
- Joint Training is the "Secret Sauce": Training a vision backbone to be more local before plugging it into an MLLM (like Locality-Aligned CLIP) helps, but it’s not optimal. SGT performs better because the vision encoder learns exactly which semantic features the LLM needs to solve the task at hand.
Key Results
SGT achieves a new state-of-the-art average score across all nine evaluated benchmarks.
- Vision-Centric Gains: On the CV-Bench, SGT improved the LLaVA 1.5 7B baseline from 55.21% to 57.85%. On RefCOCO (localization), it jumped from 77.54% to 83.17%.
- Maintaining Textual Prowess: Unlike many methods that boost vision at the expense of language, SGT retains strong performance on text-centric tasks like GQA and SciQA.
- Generalization: The method proved effective when scaling from 7B to 13B parameters and worked across different vision backbones (CLIP vs. SigLIP) and LLM architectures (Vicuna vs. Qwen).
Limitations and Future Work
The primary limitation acknowledged is computational cost. Because SGT requires a full forward and backward pass through the LLM to update the vision encoder, it is currently more suitable for fine-tuning/instruction tuning rather than massive-scale pretraining from scratch.
Future work could explore more efficient ways to distill this semantic consistency into the vision encoder or investigate if the "gold standard" reference (DINOv2) can be replaced by a more lightweight or learned guidance signal.
Summary updated