TL;DR
InternVL 1.5 is an open-source multimodal large language model (MLLM) that successfully bridges the performance gap between open-source and proprietary giants like GPT-4V. By integrating a massive 6B-parameter vision encoder, a dynamic tiling strategy for 4K resolution, and a high-quality bilingual dataset, it achieves state-of-the-art results on 8 out of 18 major benchmarks, particularly excelling in OCR and document understanding.
What Problem Does This Paper Address?
Despite the rapid advancement of open-source Multimodal Large Language Models (MLLMs), a significant "capability gap" remains when compared to proprietary models like GPT-4V or Gemini Pro. The authors identify three primary bottlenecks:
- Parameter Scale: Most open-source models use relatively small vision encoders (around 300M parameters) paired with 7B or 13B LLMs, whereas proprietary models scale much larger.
- Image Resolution: Open-source models often rely on fixed, low-resolution inputs (e.g., 336x336), which causes them to struggle with fine details in documents, charts, and small text.
- Multilingual Capability: Many open-source suites are heavily English-centric, leading to sub-optimal performance in non-English scene understanding and OCR tasks.
InternVL 1.5 aims to solve these issues to provide a truly competitive, open-source alternative for the research community.
Methods
The authors introduce three core improvements to the InternVL architecture:
- Strong Vision Encoder (InternViT-6B): Instead of using a standard off-the-shelf encoder, they refined InternViT-6B using a continuous learning strategy. This massive encoder provides a level of visual representation that rivals the linguistic power of large LLMs. Interestingly, they found these visual features are "portable"—the encoder maintained high performance even when the underlying LLM was swapped from Yi-34B to InternLM2-20B.
- Dynamic High-Resolution: To handle high-res images without massive distortion, InternVL 1.5 uses a tiling approach. It divides images into 448×448 tiles (up to 40 tiles for 4K resolution) based on the aspect ratio. It also includes a "thumbnail" view of the whole image to maintain global context.
- Architecture Scaling: The model follows the "ViT-MLP-LLM" paradigm, connecting the InternViT-6B to InternLM2-20B via an MLP projector. To keep the token count manageable at high resolutions, they use a pixel shuffle operation to reduce the number of visual tokens by three-quarters.
Datasets
The model was trained in two stages (pre-training and fine-tuning) using a massive collection of diverse data:
- Pre-training: Utilized 53.9% captioning data (Laion, COCO), 32% large-scale OCR data, and smaller portions of detection and specialized OCR datasets.
- Fine-tuning: A meticulously curated bilingual dataset covering general QA (VQAv2, GQA), science (AI2D, ScienceQA), charts (ChartQA), and mathematics.
- Data Translation Pipeline: To boost bilingual proficiency, the authors developed a pipeline using LLMs (like GPT-3.5) to translate high-quality English datasets into natural-sounding Chinese, ensuring the model doesn't just "translate" but understands cultural context.
Key Insights and Ideas
- Resolution is Task-Specific: One of the most interesting findings is that while OCR-related tasks (DocVQA, TextVQA) see massive gains from higher resolutions, general scene understanding tasks (like AI2D or MMMU) actually show a slight performance decline at extremely high tile counts. This suggests that "more resolution" isn't a silver bullet for every task.
- The VFM-LLM Synergy: The authors argue that for a large-scale LLM (e.g., 20B+), a standard 300M vision foundation model (VFM) is a bottleneck. Scaling the VFM to 6B allows the visual system to "keep up" with the reasoning capabilities of the language model.
- Zero-Shot Scaling: Although the model was trained on up to 12 tiles, it can "zero-shot" scale up to 40 tiles during inference, allowing it to process 4K images effectively without having seen them during training.
Key Results
InternVL 1.5 achieves state-of-the-art (SOTA) results in 8 out of 18 multimodal benchmarks.
- OCR Dominance: It outperforms proprietary models like GPT-4V and Gemini Pro 1.5 on OCR-heavy benchmarks including DocVQA (90.9%), ChartQA (83.8%), and TextVQA (80.6%).
- Bilingual Proficiency: Thanks to the translation pipeline, it significantly surpasses GPT-4V in Chinese-related tasks (e.g., MMBench-CN and CCBench).
- Math Reasoning: On MathVista, InternVL 1.5 (53.5%) beats GPT-4V (49.9%) and Claude-3 Opus (50.5%), showing robust logical reasoning.
Limitations and Future Work
While InternVL 1.5 closes the gap significantly, the authors acknowledge a few areas for improvement:
- Multi-turn Conversation: The model still trails GPT-4V in multi-turn dialogue benchmarks (ConvBench), likely due to the smaller scale of the language backbone compared to the massive proprietary models.
- Computational Cost: High-resolution processing results in a large number of visual tokens (up to 10,496 for 4K), which increases inference latency and memory usage.
- Language Expansion: Currently focused on English and Chinese, the authors plan to extend the data translation pipeline to support more languages in the future.
Summary updated