LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding

Jiaxing Zhao, Boyuan Sun, Xiang Chen, Xihan Wei, Qibin HouarXiv 2025 · DOI · arXiv

Abstract

LLaVA-Octopus is a novel video multimodal large language model (MLLM) designed to overcome the limitations of static, single-projector architectures. By employing an instruction-driven "router" to dynamically fuse three distinct types of visual projectors—image-based, spatial-temporal, and token-compress—the model adaptively selects the optimal visual features for specific tasks. LLaVA-Octopus achieves state-of-the-art performance across major video understanding benchmarks, including MVBench (66.9%) and VideoMME (54.7%), demonstrating superior flexibility in handling diverse visual reasoning requirements.

Introduction

In Video MLLMs, the "projector" serves as the critical bridge translating high-level visual features from an encoder (such as CLIP or SigLIP) into tokens compatible with a Large Language Model (LLM). Current models typically rely on a single projector type, which creates a bottleneck: different video tasks require different visual "diets."

Image-based projectors excel at fine-grained scene details but struggle with motion; spatial-temporal projectors capture actions but often sacrifice spatial resolution; and token-compress projectors enable long-video processing at the cost of precision. LLaVA-Octopus addresses this by moving from a static paradigm to an adaptive, multi-projector fusion approach.

Comparison of the classical paradigm versus the proposed instruction-driven adaptive projector fusion architecture.

Methodology

Instruction-Driven Adaptive Router

The core innovation of LLaVA-Octopus is the Projector Fusion Router. Instead of selecting a single projector, the model utilizes three in parallel:

  1. Image-based (MLP): Processes 8 frames with high spatial detail.
  2. Spatial-temporal (STC): Uses 3D convolutions to capture motion across those 8 frames.
  3. Token-compress: Processes a larger window of 128 frames, compressing information to maintain a manageable token count.

To integrate these features, the model performs Token Alignment, ensuring all three projectors output a consistent number of tokens (1,568). A BERT-based router analyzes the semantic meaning of the user's prompt to generate dynamic weights. These weights determine the contribution of each projector to the final fused visual embedding fed into the LLM (Qwen2.5-7B).

Training Strategy

The model undergoes a two-phase training process. First, Multi-task Pre-training uses ~2.6M image-text and video-text samples to train the projectors. Second, Instruction Tuning utilizes the Oryx dataset (631K samples) and specialized video datasets to fine-tune the router and the LLM.

Key Results and Insights

LLaVA-Octopus demonstrates significant performance gains over existing models:

  • MVBench: Achieved 66.9%, outperforming previous state-of-the-art models and significantly exceeding GPT-4V's zero-shot performance.
  • VideoMME: Reached 54.7%, confirming strong capabilities in long-form video understanding.
  • Zero-Shot QA: Consistently outperformed specialized models like VideoLLaMA2 and LLaMA-VID on MSVD-QA and ActivityNet-QA.
Qualitative comparison of video question answering performance across different MLLM architectures.

Projector Complementarity

Ablation studies confirm that the projectors are complementary rather than redundant. For scene-detail queries, the router prioritizes the image-based projector (avg. weight 0.71), while for dynamic counting tasks requiring long-term context, it favors the token-compress projector (avg. weight 0.61). This confirms that a diverse projector setup is more effective than simply scaling a single projector type.

Conclusion

LLaVA-Octopus proves that instruction-driven fusion of specialized projectors is a powerful strategy for video understanding. While the parallel architecture increases initial computational overhead, the resulting gains in task-specific accuracy and flexibility represent a significant step forward for Video MLLMs. Future work may involve designing novel projector architectures specifically optimized for this fusion paradigm.

Summary updated

Abstract

In this paper, we introduce LLaVA-Octopus, a novel video multimodal large language model. LLaVA-Octopus adaptively weights features from different visual projectors based on user instructions, enabling us to leverage the complementary strengths of each projector. We observe that different visual projectors exhibit distinct characteristics when handling specific tasks. For instance, some projectors excel at capturing static details, while others are more effective at processing temporal information, and some are better suited for tasks requiring temporal coherence. By dynamically adjusting feature weights according to user instructions, LLaVA-Octopus dynamically selects and combines the most suitable features, significantly enhancing the model's performance in multimodal tasks. Experimental results demonstrate that LLaVA-Octopus achieves excellent performance across multiple benchmarks, especially in tasks such as video question answering, long video understanding, and comprehensive multi-choices benchmarks, highlighting its broad application potential.
BibTeX
@misc{zhao2025llava,
  author = {Jiaxing Zhao and Boyuan Sun and Xiang Chen and Xihan Wei and Qibin Hou},
  title = {{LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding}},
  year = {2025},
  eprint = {2501.05067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.05067},
}
Figure