Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models

Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Liang Wang, Rong JinarXiv 2024 · OpenReview

Abstract

Seeing clearly with high resolution is a foundation of Large Multimodal Models (LMMs), which has been proven to be vital for visual perception and reasoning. Existing works usually employ a straightforward resolution upscaling method, where the image consists of global and local branches, with the latter being the sliced image patches but resized to the same resolution as the former. This means that higher resolution requires more local patches, resulting in exorbitant computational expenses, and meanwhile, the dominance of local image tokens may diminish the global context. In this paper, we dive into the problems and propose a new framework as well as an elaborate optimization strategy. Specifically, we extract contextual information from the global view using a mixture of adapters, based on the observation that different adapters excel at different tasks. With regard to local patches, learnable query embeddings are introduced to reduce image tokens, the most important tokens accounting for the user question will be further selected by a similarity-based selector. Our empirical results demonstrate a 'less is more' pattern, where utilizing fewer but more informative local image tokens leads to improved performance. Besides, a significant challenge lies in the training strategy, as simultaneous end-to-end training of the global mining block and local compression block does not yield optimal results. We thus advocate for an alternating training way, ensuring balanced learning between global and local aspects. Finally, we also introduce a challenging dataset with high requirements for image detail, enhancing the training of the local compression layer. The proposed method, termed LMM with Sophisticated Tasks, Local image compression, and Mixture of global Experts (SliME), achieves leading performance across various benchmarks with only 2 million training data.

Introduction

Large Multimodal Models (LMMs) like LLaVA have a "vision problem." To understand fine details—such as small text or complex charts—they require high-resolution input. The standard approach involves slicing a high-resolution image into many smaller patches and feeding them all into the model. However, this creates two major issues: computational overload due to token explosion and context dilution, where the "big picture" (global context) is drowned out by thousands of local patch tokens.

To address these challenges, we introduce SliME (Sophisticated Tasks, Local image compression, and Mixture of global Experts), a framework designed to keep high-resolution details without breaking the computational budget or losing global context.

The SliME Framework

The SliME architecture relies on three core innovations to optimize high-resolution processing:

  1. Global Mixture of Experts (MoE): Instead of a single adapter, SliME uses a mixture of adapters (e.g., a simple MLP and a Q-former). Since different adapters excel at different tasks, "softly mixing" their outputs provides a richer global representation.
  2. Local Feature Compression & Selection: For high-resolution patches, SliME introduces learnable query embeddings to compress the data. A Text-Guided Router then selects only the tokens most relevant to the user's specific question.
  3. The "Less is More" Pattern: Empirical results show that filtering out irrelevant background noise through the text-guided router allows the model to focus better on the task, proving that fewer, more informative tokens can outperform a massive, unfiltered token set.
The SliME architecture featuring the Text-Guided Router and local feature compression.

Alternating Training Strategy

A significant discovery in our research is the "bilinear optimization trap." Simultaneous end-to-end training of global and local components often fails to converge optimally because the model tends to over-rely on easier-to-learn global features. To solve this, we advocate for a three-stage alternating strategy:

  1. Stage 1: Train the global mining block.
  2. Stage 2: Fix the global block and train the local compression layer.
  3. Stage 3: Fine-tune both components together.

Experimental Results

Despite being trained on only 2 million samples—significantly less than many competitors—SliME-8B (based on Llama-3) achieved leading performance across 15 benchmarks, including MMMU, MME, and ScienceQA. Notably, SliME-8B outperformed Gemini Pro on MMBench (75.0 vs 73.6) and MME (1578 vs 1496).

Radar chart comparing SliME variants across multiple benchmarks, highlighting the effectiveness of alternating training.

To further push reasoning limits, we introduced the SMR (Science and Mathematical Reasoning) dataset, an amalgamation of nine challenging sources requiring profound visual detail understanding, such as Arxiv-QA and ChartVQA.

Conclusion

SliME demonstrates that high-resolution LMMs can be both leaner and smarter. By prioritizing informative token selection and employing a strategic alternating training scheme, SliME achieves state-of-the-art reasoning capabilities with high efficiency. Future work will explore further token reduction techniques like token merging to minimize the computational footprint even more.

Summary updated

Abstract

Seeing clearly with high resolution is a foundation of Large Multimodal Models (LMMs), which has been proven to be vital for visual perception and reasoning. Existing works usually employ a straightforward resolution upscaling method, where the image consists of global and local branches, with the latter being the sliced image patches but resized to the same resolution as the former. This means that higher resolution requires more local patches, resulting in exorbitant computational expenses, and meanwhile, the dominance of local image tokens may diminish the global context. In this paper, we dive into the problems and propose a new framework as well as an elaborate optimization strategy. Specifically, we extract contextual information from the global view using a mixture of adapters, based on the observation that different adapters excel at different tasks. With regard to local patches, learnable query embeddings are introduced to reduce image tokens, the most important tokens accounting for the user question will be further selected by a similarity-based selector. Our empirical results demonstrate a `less is more' pattern, where \textit{utilizing fewer but more informative local image tokens leads to improved performance}. Besides, a significant challenge lies in the training strategy, as simultaneous end-to-end training of the global mining block and local compression block does not yield optimal results. We thus advocate for an alternating training way, ensuring balanced learning between global and local aspects. Finally, we also introduce a challenging dataset with high requirements for image detail, enhancing the training of the local compression layer. The proposed method, termed LMM with Sophisticated Tasks, Local image compression, and Mixture of global Experts (SliME), achieves leading performance across various benchmarks with only 2 million training data.
BibTeX
@misc{zhang2024beyond,
  author = {Yi-Fan Zhang and Qingsong Wen and Chaoyou Fu and Xue Wang and Zhang Zhang and Liang Wang and Rong Jin},
  title = {{Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models}},
  year = {2024},
  eprint = {2406.08487},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.08487},
}
Figure