Abstract
LLaVA-Octopus is a novel video multimodal large language model (MLLM) designed to overcome the limitations of static, single-projector architectures. By employing an instruction-driven "router" to dynamically fuse three distinct types of visual projectors—image-based, spatial-temporal, and token-compress—the model adaptively selects the optimal visual features for specific tasks. LLaVA-Octopus achieves state-of-the-art performance across major video understanding benchmarks, including MVBench (66.9%) and VideoMME (54.7%), demonstrating superior flexibility in handling diverse visual reasoning requirements.
Introduction
In Video MLLMs, the "projector" serves as the critical bridge translating high-level visual features from an encoder (such as CLIP or SigLIP) into tokens compatible with a Large Language Model (LLM). Current models typically rely on a single projector type, which creates a bottleneck: different video tasks require different visual "diets."
Image-based projectors excel at fine-grained scene details but struggle with motion; spatial-temporal projectors capture actions but often sacrifice spatial resolution; and token-compress projectors enable long-video processing at the cost of precision. LLaVA-Octopus addresses this by moving from a static paradigm to an adaptive, multi-projector fusion approach.
Methodology
Instruction-Driven Adaptive Router
The core innovation of LLaVA-Octopus is the Projector Fusion Router. Instead of selecting a single projector, the model utilizes three in parallel:
- Image-based (MLP): Processes 8 frames with high spatial detail.
- Spatial-temporal (STC): Uses 3D convolutions to capture motion across those 8 frames.
- Token-compress: Processes a larger window of 128 frames, compressing information to maintain a manageable token count.
To integrate these features, the model performs Token Alignment, ensuring all three projectors output a consistent number of tokens (1,568). A BERT-based router analyzes the semantic meaning of the user's prompt to generate dynamic weights. These weights determine the contribution of each projector to the final fused visual embedding fed into the LLM (Qwen2.5-7B).
Training Strategy
The model undergoes a two-phase training process. First, Multi-task Pre-training uses ~2.6M image-text and video-text samples to train the projectors. Second, Instruction Tuning utilizes the Oryx dataset (631K samples) and specialized video datasets to fine-tune the router and the LLM.
Key Results and Insights
LLaVA-Octopus demonstrates significant performance gains over existing models:
- MVBench: Achieved 66.9%, outperforming previous state-of-the-art models and significantly exceeding GPT-4V's zero-shot performance.
- VideoMME: Reached 54.7%, confirming strong capabilities in long-form video understanding.
- Zero-Shot QA: Consistently outperformed specialized models like VideoLLaMA2 and LLaMA-VID on MSVD-QA and ActivityNet-QA.
Projector Complementarity
Ablation studies confirm that the projectors are complementary rather than redundant. For scene-detail queries, the router prioritizes the image-based projector (avg. weight 0.71), while for dynamic counting tasks requiring long-term context, it favors the token-compress projector (avg. weight 0.61). This confirms that a diverse projector setup is more effective than simply scaling a single projector type.
Conclusion
LLaVA-Octopus proves that instruction-driven fusion of specialized projectors is a powerful strategy for video understanding. While the parallel architecture increases initial computational overhead, the resulting gains in task-specific accuracy and flexibility represent a significant step forward for Video MLLMs. Future work may involve designing novel projector architectures specifically optimized for this fusion paradigm.
Summary updated