Introduction: Overcoming Visual Myopia
Current Large Vision-Language Models (LVLMs) often suffer from "visual myopia," typically capped at resolutions around 1500 × 1500 pixels. This limitation leads to a significant loss of fine-grained details when processing high-resolution infographics, dense scientific tables, or 4K website screenshots. InternLM-XComposer2-4KHD addresses this by supporting image inputs up to 4K HD (3840 × 1600) and beyond, ensuring structural integrity and detail preservation through a novel dynamic patching strategy.
Methodology: Dynamic Partitioning and Spatial Anchoring
The researchers introduced several key innovations to the standard patch-division paradigm to handle varying aspect ratios and high resolutions without excessive computational overhead.
Dynamic Image Partition
Instead of standard square resizing, the model employs a "best-fit" layout of 336 × 336 patches (the native resolution of the ViT encoder). It calculates an optimal grid ($p_w \times p_h$) that preserves the original aspect ratio while staying within a defined token budget.
Global-Local Format
Each image is processed through two distinct views:
- Global View: A resized 336 × 336 version providing macro-level context.
- Local View: High-resolution patches that capture fine details necessary for OCR and data interpretation.
2D Structure Newline Indicator
To prevent spatial information loss when flattening 2D patches into a 1D sequence for the Large Language Model (LLM), the authors added a learnable newline (\n) token at the end of each row of patches. This acts as a spatial anchor, helping the LLM "understand" the layout of the high-resolution grid.
Training and Datasets
The 7B-parameter model, utilizing InternLM2-7B as the language backbone and OpenAI ViT-Large/14 as the vision encoder, underwent a two-stage training process. Pre-training focused on general semantic alignment using massive datasets like ShareGPT4V-PT and LAION400M. This was followed by Supervised Fine-Tuning (SFT) on 16 benchmarks, with a heavy emphasis on HD-OCR tasks such as DocVQA, ChartQA, and InfoVQA.
Key Results and Insights
InternLM-XComposer2-4KHD demonstrates that resolution scaling has not yet hit a ceiling; increasing training resolution consistently leads to performance gains. Interestingly, the model shows "zero-shot" resolution scaling, achieving better results at resolutions higher than those used during training for text-heavy tasks.
Despite its relatively small 7B-parameter size, the model matches or surpasses giants like GPT-4V and Gemini Pro in 10 out of 16 benchmarks. It achieved State-of-the-Art (SOTA) results on 6 benchmarks, including:
- 90.0% on DocVQA
- 81.0% on ChartQA
- 68.6% on InfoVQA (surpassing the previous open-source SOTA by nearly 18%)
Conclusion and Future Work
While the quadratic growth of visual tokens presents a computational challenge, InternLM-XComposer2-4KHD proves that high-resolution processing is vital for the next generation of LVLMs. Future research will likely focus on more efficient token-merging strategies to push resolutions toward 8K and beyond without a linear increase in compute requirements.
Summary updated