Introduction
Multimodal Large Language Models (MLLMs) have achieved remarkable success in understanding visual content. However, as these models scale to handle high-resolution images, they encounter a significant bottleneck: the "token explosion." High-resolution processing often involves partitioning images into numerous sub-images, resulting in thousands of visual tokens. Because the computational complexity of the Large Language Model (LLM) attention mechanism grows quadratically with sequence length, this surge in tokens leads to massive memory overhead and sluggish inference speeds.
LLaVA-Meteor introduces a novel Top-Down Compression strategy designed to mitigate this issue. By slashing the number of visual tokens sent to the LLM by up to 95% without sacrificing performance, LLaVA-Meteor achieves state-of-the-art results on high-resolution vision-language tasks while significantly boosting efficiency.
The Challenge of Token Explosion
Existing methods to manage visual token counts generally fall into two categories: Fusion and Selection. Fusion techniques, such as PixelShuffle, merge tokens to reduce their number but often destroy fine-grained spatial structures. Selection techniques pick "important" tokens based on saliency but frequently ignore the specific task or instruction, potentially discarding tokens critical for answering a user's query (e.g., small text in the background).
Methodology: Top-Down Compression
LLaVA-Meteor bridges the gap between fusion and selection through a two-stage paradigm that ensures global awareness and instruction sensitivity.
1. Flash Global Fusion (FGF)
The authors argue that global context is a prerequisite for effective selection; a model cannot decide which tokens to discard until every token "knows" the context of the entire image. To achieve this efficiently, LLaVA-Meteor utilizes State Space Models (SSMs/Mamba), which offer linear complexity compared to the quadratic cost of Transformers. To address Mamba's inherent lack of 2D spatial bias, the model employs a local-to-single scanning strategy, where each token aggregates information from its 3x3 neighborhood before the global scan. Additionally, a learnable Instruction Token (INS) is inserted into the sequence to aggregate task-relevant features during this fusion stage.
2. Visual-Native Selection (VNS)
Following global fusion, the model employs a dual-expert filtering system to select the most relevant tokens:
- The Visual Expert: Utilizes attention scores from the vision encoder’s class token to identify general saliency (e.g., identifying the main subject of an image).
- The Native Expert: Calculates the similarity between the fused visual tokens and the specialized Instruction Token. This ensures that tokens relevant to the specific user prompt (e.g., specific text or small details) are preserved even if they are not globally salient.
The final set of tokens is chosen based on a weighted average of these two experts, allowing for a highly compressed yet informative representation.
Experimental Results
LLaVA-Meteor was evaluated across 12 benchmarks, including text-oriented VQA (TextVQA, DocVQA) and general VQA (GQA, VQAv2). The results demonstrate significant improvements in both efficiency and performance:
- Efficiency: The model reduces visual tokens by 75% to 95%. Even when compressed to just 32 tokens per sub-image (down from 144+), it remains highly competitive.
- Performance: At the same token budget as LLaVA-UHD, LLaVA-Meteor improved average performance by +2.0 points across all benchmarks, reaching new state-of-the-art levels on VQAT (69.9) and POPE (89.9).
- Speed: Due to shorter sequences, the model achieved higher Tokens Per Second (TPS) and reduced training time (e.g., 12.1 hours compared to 17.2 hours for LLaVA-UHD).
Conclusion
LLaVA-Meteor demonstrates that "less is more" in visual instruction tuning, provided the reduction is handled intelligently. By performing global fusion before selection and incorporating instruction-aware experts, the Top-Down Compression paradigm allows MLLMs to maintain high performance on complex, high-resolution tasks while drastically reducing the computational burden of the LLM backbone.
Summary updated