InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, Jiaqi Wang

Introduction: Overcoming Visual Myopia

Current Large Vision-Language Models (LVLMs) often suffer from "visual myopia," typically capped at resolutions around 1500 × 1500 pixels. This limitation leads to a significant loss of fine-grained details when processing high-resolution infographics, dense scientific tables, or 4K website screenshots. InternLM-XComposer2-4KHD addresses this by supporting image inputs up to 4K HD (3840 × 1600) and beyond, ensuring structural integrity and detail preservation through a novel dynamic patching strategy.

The model handles high-resolution images by partitioning them into patches, allowing for the comprehension of fine details in complex charts.

Methodology: Dynamic Partitioning and Spatial Anchoring

The researchers introduced several key innovations to the standard patch-division paradigm to handle varying aspect ratios and high resolutions without excessive computational overhead.

Dynamic Image Partition

Instead of standard square resizing, the model employs a "best-fit" layout of 336 × 336 patches (the native resolution of the ViT encoder). It calculates an optimal grid ($p_w \times p_h$) that preserves the original aspect ratio while staying within a defined token budget.

Global-Local Format

Each image is processed through two distinct views:

  1. Global View: A resized 336 × 336 version providing macro-level context.
  2. Local View: High-resolution patches that capture fine details necessary for OCR and data interpretation.

2D Structure Newline Indicator

To prevent spatial information loss when flattening 2D patches into a 1D sequence for the Large Language Model (LLM), the authors added a learnable newline (\n) token at the end of each row of patches. This acts as a spatial anchor, helping the LLM "understand" the layout of the high-resolution grid.

Training and Datasets

The 7B-parameter model, utilizing InternLM2-7B as the language backbone and OpenAI ViT-Large/14 as the vision encoder, underwent a two-stage training process. Pre-training focused on general semantic alignment using massive datasets like ShareGPT4V-PT and LAION400M. This was followed by Supervised Fine-Tuning (SFT) on 16 benchmarks, with a heavy emphasis on HD-OCR tasks such as DocVQA, ChartQA, and InfoVQA.

Key Results and Insights

InternLM-XComposer2-4KHD demonstrates that resolution scaling has not yet hit a ceiling; increasing training resolution consistently leads to performance gains. Interestingly, the model shows "zero-shot" resolution scaling, achieving better results at resolutions higher than those used during training for text-heavy tasks.

Radar chart showing InternLM-XComposer2-4KHD matching or outperforming GPT-4V and Gemini Pro across nearly two-thirds of evaluated benchmarks.

Despite its relatively small 7B-parameter size, the model matches or surpasses giants like GPT-4V and Gemini Pro in 10 out of 16 benchmarks. It achieved State-of-the-Art (SOTA) results on 6 benchmarks, including:

  • 90.0% on DocVQA
  • 81.0% on ChartQA
  • 68.6% on InfoVQA (surpassing the previous open-source SOTA by nearly 18%)

Conclusion and Future Work

While the quadratic growth of visual tokens presents a computational challenge, InternLM-XComposer2-4KHD proves that high-resolution processing is vital for the next generation of LVLMs. Future research will likely focus on more efficient token-merging strategies to push resolutions toward 8K and beyond without a linear increase in compute requirements.

Summary updated

BibTeX
@inproceedings{dong2024internlm,
  author = {Xiaoyi Dong and Pan Zhang and Yuhang Zang and Yuhang Cao and Bin Wang and Linke Ouyang and Songyang Zhang and Haodong Duan and Wenwei Zhang and Yining Li and Hang Yan and Yang Gao and Zhe Chen and Xinyue Zhang and Wei Li and Jingwen Li and Wenhai Wang and Kai Chen and Conghui He and Xingcheng Zhang and Jifeng Dai and Yu Qiao and Dahua Lin and Jiaqi Wang},
  title = {{InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=nRp0XhTf61},
}
Figure