How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, Wenhai Wang

TL;DR

InternVL 1.5 is an open-source multimodal large language model (MLLM) that successfully bridges the performance gap between open-source and proprietary giants like GPT-4V. By integrating a massive 6B-parameter vision encoder, a dynamic tiling strategy for 4K resolution, and a high-quality bilingual dataset, it achieves state-of-the-art results on 8 out of 18 major benchmarks, particularly excelling in OCR and document understanding.

What Problem Does This Paper Address?

Despite the rapid advancement of open-source Multimodal Large Language Models (MLLMs), a significant "capability gap" remains when compared to proprietary models like GPT-4V or Gemini Pro. The authors identify three primary bottlenecks:

  1. Parameter Scale: Most open-source models use relatively small vision encoders (around 300M parameters) paired with 7B or 13B LLMs, whereas proprietary models scale much larger.
  2. Image Resolution: Open-source models often rely on fixed, low-resolution inputs (e.g., 336x336), which causes them to struggle with fine details in documents, charts, and small text.
  3. Multilingual Capability: Many open-source suites are heavily English-centric, leading to sub-optimal performance in non-English scene understanding and OCR tasks.

InternVL 1.5 aims to solve these issues to provide a truly competitive, open-source alternative for the research community.

Methods

The authors introduce three core improvements to the InternVL architecture:

  • Strong Vision Encoder (InternViT-6B): Instead of using a standard off-the-shelf encoder, they refined InternViT-6B using a continuous learning strategy. This massive encoder provides a level of visual representation that rivals the linguistic power of large LLMs. Interestingly, they found these visual features are "portable"—the encoder maintained high performance even when the underlying LLM was swapped from Yi-34B to InternLM2-20B.
  • Dynamic High-Resolution: To handle high-res images without massive distortion, InternVL 1.5 uses a tiling approach. It divides images into 448×448 tiles (up to 40 tiles for 4K resolution) based on the aspect ratio. It also includes a "thumbnail" view of the whole image to maintain global context.
  • Architecture Scaling: The model follows the "ViT-MLP-LLM" paradigm, connecting the InternViT-6B to InternLM2-20B via an MLP projector. To keep the token count manageable at high resolutions, they use a pixel shuffle operation to reduce the number of visual tokens by three-quarters.
Figure 3. Overall Architecture. InternVL 1.5 adopts the ViT- MLP-LLM architecture similar to popular MLLMs [62, 64], com- bining a pre-trained InternViT-6B [18] with InternLM2-20B [11] through a MLP projector. Here, we employ a simple pixel shuffle to reduce the number of visual tokens to one-quarter.

Datasets

The model was trained in two stages (pre-training and fine-tuning) using a massive collection of diverse data:

  • Pre-training: Utilized 53.9% captioning data (Laion, COCO), 32% large-scale OCR data, and smaller portions of detection and specialized OCR datasets.
  • Fine-tuning: A meticulously curated bilingual dataset covering general QA (VQAv2, GQA), science (AI2D, ScienceQA), charts (ChartQA), and mathematics.
  • Data Translation Pipeline: To boost bilingual proficiency, the authors developed a pipeline using LLMs (like GPT-3.5) to translate high-quality English datasets into natural-sounding Chinese, ensuring the model doesn't just "translate" but understands cultural context.
A summary of the diverse datasets used for pre-training and fine-tuning.

Key Insights and Ideas

  • Resolution is Task-Specific: One of the most interesting findings is that while OCR-related tasks (DocVQA, TextVQA) see massive gains from higher resolutions, general scene understanding tasks (like AI2D or MMMU) actually show a slight performance decline at extremely high tile counts. This suggests that "more resolution" isn't a silver bullet for every task.
  • The VFM-LLM Synergy: The authors argue that for a large-scale LLM (e.g., 20B+), a standard 300M vision foundation model (VFM) is a bottleneck. Scaling the VFM to 6B allows the visual system to "keep up" with the reasoning capabilities of the language model.
  • Zero-Shot Scaling: Although the model was trained on up to 12 tiles, it can "zero-shot" scale up to 40 tiles during inference, allowing it to process 4K images effectively without having seen them during training.
Figure 6. Comparison of InternVL 1.5 performance across different image resolutions. The X-axis represents the number of tiles, while the Y-axis indicates benchmark performance. The highest value and its corresponding number of tiles are highlighted. The scores of MME [26] and OCRBench [67] have been normalized to a maximum score of 100. We found that although only 1 to 12 tiles were used during training, it is possible to zero-shot scale up to 40 tiles (i.e., 4K resolution) during testing. Note that since MMMU [135] includes multiple images per sample, it may run out of memory when the number of tiles is large. Therefore, we only tested up to 18 tiles maximum, and MMMU was not included when calculating the average score.

Key Results

InternVL 1.5 achieves state-of-the-art (SOTA) results in 8 out of 18 multimodal benchmarks.

  • OCR Dominance: It outperforms proprietary models like GPT-4V and Gemini Pro 1.5 on OCR-heavy benchmarks including DocVQA (90.9%), ChartQA (83.8%), and TextVQA (80.6%).
  • Bilingual Proficiency: Thanks to the translation pipeline, it significantly surpasses GPT-4V in Chinese-related tasks (e.g., MMBench-CN and CCBench).
  • Math Reasoning: On MathVista, InternVL 1.5 (53.5%) beats GPT-4V (49.9%) and Claude-3 Opus (50.5%), showing robust logical reasoning.
Comparison of InternVL 1.5 against leading proprietary and open-source models.

Limitations and Future Work

While InternVL 1.5 closes the gap significantly, the authors acknowledge a few areas for improvement:

  • Multi-turn Conversation: The model still trails GPT-4V in multi-turn dialogue benchmarks (ConvBench), likely due to the smaller scale of the language backbone compared to the massive proprietary models.
  • Computational Cost: High-resolution processing results in a large number of visual tokens (up to 10,496 for 4K), which increases inference latency and memory usage.
  • Language Expansion: Currently focused on English and Chinese, the authors plan to extend the data translation pipeline to support more languages in the future.

Summary updated

Abstract

In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements: (1) Strong Vision Encoder: we explored a continuous learning strategy for the large-scale vision foundation model -- InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic High-Resolution: we divide images into tiles ranging from 1 to 40 of 448$\times$448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in OCR- and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL.
BibTeX
@article{chen2024how,
  author = {Zhe Chen and Weiyun Wang and Hao Tian and Shenglong Ye and Zhangwei Gao and Erfei Cui and Wenwen Tong and Kongzhi Hu and Jiapeng Luo and Zheng Ma and Ji Ma and Jiaqi Wang and Xiaoyi Dong and Hang Yan and Hewei Guo and Conghui He and Botian Shi and Zhenjiang Jin and Chao Xu and Bin Wang and Xingjian Wei and Wei Li and Wenjian Zhang and Bo Zhang and Pinlong Cai and Licheng Wen and Xiangchao Yan and Min Dou and Lewei Lu and Xizhou Zhu and Tong Lu and Dahua Lin and Yu Qiao and Jifeng Dai and Wenhai Wang},
  title = {{How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites}},
  journal = {Science China Information Sciences},
  year = {2024},
  url = {https://openreview.net/forum?id=kmo6QDY3Bb},
}
Figure