SigLIP-HD by Fine-to-Coarse Supervision

Lihe Yang, Zhen Zhao, Hengshuang Zhao

High-quality visual representation is a long-standing pursuit in computer vision. This paper introduces SigLIP-HD, a method designed to significantly boost the visual perception of Multimodal Large Language Models (MLLMs) without increasing their computational cost during inference.

TL;DR

SigLIP-HD employs a "fine-to-coarse" supervision strategy to train a standard-resolution vision encoder to mimic the features of a high-resolution "teacher." The result is a drop-in replacement for the popular SigLIP 2 encoder that dramatically improves OCR and document understanding performance at zero additional inference cost.

The Resolution Dilemma in MLLMs

In the realm of MLLMs (like GPT-4o or LLaVA), resolution is king. To understand a complex chart or read small text, models typically require high-resolution input. However, simply adding more pixels is computationally expensive. High-resolution images result in a massive increase in visual tokens, which slows down the LLM and complicates the model architecture—often requiring the image to be sliced into "tiles" and processed in multiple forward passes.

The perception gap at medium resolutions

The authors pose a provocative question: Have we truly unlocked a model's full perception capability at standard resolutions? Humans can often read text or understand scenes at "medium" resolutions (e.g., 512px) where current AI models struggle. SigLIP-HD aims to close this "perception gap" by making standard-resolution tokens inherently more informative.

Fine-to-Coarse Supervision

The core innovation of this work is a highly simple Fine-to-Coarse Supervision mechanism. The training setup involves two distinct branches:

SigLIP-HD Fine-to-Coarse Supervision Architecture
  1. The Inference (Teacher) Branch: A frozen, pre-trained SigLIP 2 encoder receives the image at two scales: a "base" global scale ($512^2$) and a "high-res" local scale ($1024^2$). These features are fused to create a high-quality "target" representation. Specifically, the high-res local features are re-assembled, interpolated, and then averaged with the base global features.
  2. The Trainable (Student) Branch: The SigLIP-HD model receives only the base-scale ($512^2$) image.

The Goal: The student branch is trained to minimize the difference (using a simple L1 loss) between its output tokens and the high-quality fused tokens from the teacher.

By forcing the student to "mimic" the features that would normally only be available at higher resolutions, the model learns to encode fine-grained details into its standard-resolution tokens. Crucially, once training is finished, the teacher branch is discarded. The final SigLIP-HD model has the exact same architecture and token count as the original SigLIP 2, resulting in a zero-overhead upgrade.

Datasets and Evaluation

  • Training: The model was trained on 4.5 million raw images from the Cambrian-1 dataset, which includes a diverse mix of natural scenes, documents, and web content.
  • Evaluation: The authors tested the model across a massive battery of MLLM benchmarks, including OCR & Chart tasks (DocVQA, ChartQA, TextVQA, InfoVQA), General Perception (MME, POPE, MMBench, GQA), and Knowledge tasks (ScienceQA, AI2D).

Key Insights and Ideas

  • Multi-scale is better than just High-res: The authors found that a teacher seeing both $512^2$ and $1024^2$ images produced better features than a teacher seeing only a $1024^2$ image. The lower resolution provides global context that helps "anchor" the high-resolution details.
  • Simplicity Wins: In their ablation studies, the most effective way to fuse features was simple linear interpolation and averaging, and the most effective loss was a straightforward L1 loss. More complex methods like "pixel unshuffle" or cosine similarity actually performed worse.
  • Universality: The method isn't tied exclusively to SigLIP. The authors demonstrated that the same fine-to-coarse supervision could improve the aging OpenAI-CLIP model, suggesting this is a general property of vision-language encoders.

Key Results

SigLIP-HD consistently outperformed the baseline SigLIP 2 across various LLM backbones (Vicuna, Llama-3.2, Qwen2.5):

  • OCR Gains: On the LLaVA-1.5 protocol, SigLIP-HD improved DocVQA scores from 32.2 to 34.7 and ChartQA from 19.3 to 20.2—all while using the exact same number of tokens.
  • Compatibility with AnyRes: Even when used within "AnyRes" frameworks (which already use high-res tiling), SigLIP-HD provided a boost. For example, it improved ChartQA from 63.9 to 67.4 (+3.5 points) in a high-res setting.
  • Efficiency: The model was trained in just 34 hours on 32 A100 GPUs, making it a relatively "cheap" enhancement compared to pre-training a new vision encoder from scratch.

Limitations and Future Work

While SigLIP-HD significantly polishes visual representations, the authors acknowledge it cannot entirely replace high-resolution inputs for every task. Some extremely fine details are physically lost during downsampling and cannot be recovered by the encoder alone.

Future work could explore applying this supervision to even larger vision encoders (e.g., SigLIP-G), investigating if "coarse-to-fine" or multi-teacher setups could further enhance the feature space, and optimizing the fusion weights for specific domains like medical imaging or satellite data.

Summary updated

Abstract

High-quality visual representation is a long-standing pursuit in computer vision. In the context of multimodal LLMs (MLLMs), feeding higher-resolution images can produce more fine-grained visual tokens. However, it introduces additional computational and design complexity, due to multiple forward passes and post-processing of increased tokens. Before simply adopting a higher resolution, have we truly unlocked the model's full perception capability at a standard resolution? Therefore, we study an interesting problem: how to achieve fine visual perception under lower cost without larger images. We present SigLIP-HD in this work. The core is a highly simple fine-to-coarse supervision design. We enforce the coarse feature of a mid-resolution image to mimic the fine-grained feature of its high-resolution version. We build this framework on the advanced SigLIP 2 model. Our final model produces better visual tokens at exactly the same inference budget. It is validated on extensive MLLM benchmarks and consistently delivers stronger results than our baseline model, especially on OCR-related tasks.
BibTeX
@inproceedings{yang2026siglip,
  author = {Lihe Yang and Zhen Zhao and Hengshuang Zhao},
  title = {{SigLIP-HD by Fine-to-Coarse Supervision}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=XeLrfKEOZS},
}
Figure