Native Resolution Encoding for Vision-Language Models

This direction is merged into another direction: aspect ratio distortion. Its last version stays readable and citable here.

Version 1 of 151 papers68 conceptsfrozen 2026-10-04

For years, the standard approach in computer vision was to force every image into a fixed-size square, typically 224x224 or 336x336 pixels. While this "square-resize" paradigm worked for identifying dogs or trees in natural photos, it proved disastrous for "visually-situated language"—the text found in documents, user interfaces (UIs), and infographics.

When you squash a tall, thin receipt or a wide panoramic screenshot into a small square, the text becomes distorted, characters blur together, and the spatial relationships between elements are destroyed. To solve this, a new generation of Vision-Language Models (VLMs) has moved toward Native Resolution Encoding, a strategy that processes images at their original aspect ratios and scales.

The Resolution Dilemma

The fundamental problem with fixed-resolution resizing is information loss. As noted by Junbo Niu and colleagues in Native Visual Understanding, forcing diverse real-world images into low-resolution squares creates a "Resolution Dilemma" where fine details are blurred beyond recognition.

In document analysis, the relative height and width of a page provide critical structural cues. Gabriela Csurka and the team at Xerox Research observed as early as 2016 that preserving the original aspect ratio is crucial for document understanding. If a model squashes a vertical document into a square, a "0" might look like an "O," and a "1" might disappear entirely.

Preserving Geometric Integrity

The core intuition behind native resolution is that geometric integrity matters more than a fixed pixel count. Instead of resizing, models like Pix2Struct, developed by Kenton Lee and others, scale images to fit a maximum number of patches while strictly maintaining their original aspect ratio.

This approach ensures that:

  1. Characters remain legible: Letters are not stretched or squashed, keeping them within the optimal recognition range of the vision encoder.
  2. Spatial layouts are preserved: In UIs and forms, the distance between a label and its input box is a vital signal. Native encoding keeps these distances consistent.
  3. Small features survive: As Wei Bai demonstrates in CMERNet, preserving native proportions is essential for recognizing tiny, high-information features like mathematical subscripts and diacritics.

Dynamic Tiling and Variable Patching

To implement this without breaking the Transformer architecture—which typically expects a fixed sequence length—researchers use two primary mechanisms: Dynamic Tiling and Variable-Grid Patching.

In the ScreenAI model, Google researchers introduced a variable-grid patching strategy that scales the input grid to match the image's original shape. Similarly, DeepSeek-VL2 uses a dynamic tiling strategy that adapts to the aspect ratio, allowing the model to interpret dense text in charts or panoramas without the distortions of fixed-square resizing.

Models like Pixtral 12B by Pravesh Agrawal and Qwen2-VL by Chang Zhou take this further by using "Naive Dynamic Resolution." This allows the model to process images of any size by decomposing them into a variable number of visual tokens. By avoiding arbitrary tiling, these models maintain a global context that is often lost when an image is chopped into independent sub-tiles.

Spatial Awareness with 2D Embeddings

Processing images of arbitrary shapes requires the model to know where each patch came from in the original image. Standard 1D positional embeddings (which assume a fixed sequence) are insufficient for this.

To solve this, models like Qwen2-VL implement Multimodal Rotary Position Embedding (M-RoPE), while Seed1.5-VL by Dong Guo utilizes 2D Rotary Positional Embeddings. These mechanisms allow the model to understand the 2D coordinates of each patch regardless of whether the input is a long vertical receipt or a wide website banner.

Why It Works: The Evidence

The shift toward native resolution has led to a dramatic leap in performance for compact models. POINTS1.5, a 7B-parameter model from Tencent, uses a NaViT-style dynamic resolution to outperform models several times its size in OCR and mathematical reasoning.

Similarly, Hugo Laurençon and the Hugging Face team showed in Idefics2 that preserving the original aspect ratio is one of the most impactful design decisions in the VLM pipeline. By combining this with document-specific pre-training, they created a model that punches significantly above its weight class in structural document analysis.

By treating images as they are—rather than as the squares we want them to be—native resolution encoding allows models to see the world with the same spatial fidelity as the humans who designed the documents and interfaces they are trying to understand.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_native_resolution_encoding_for_vision_la_3274af59,
  title = {Native Resolution Encoding for Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/native-resolution-encoding-for-vision-language-models/rdr_3274af5936b746f6b48fd253e20b5354},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_native_resolution_encoding_for_vision_la_3274af59_v1,
  title = {Native Resolution Encoding for Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/native-resolution-encoding-for-vision-language-models/rdr_3274af5936b746f6b48fd253e20b5354/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}