From Image to Video, what do we need in multimodal LLMs?

Suyuan Huang, Haoxin Zhang, Yan Gao, Yao Hu, Zengchang QinarXiv 2024 · DOI · arXiv

Introduction

The rapid advancement of Multimodal Large Language Models (MLLMs) has primarily focused on static images. Extending these capabilities to video—which requires understanding complex temporal dynamics—has traditionally been a resource-heavy endeavor. Researchers have recently introduced RED-VILLM, a Resource-Efficient Development pipeline designed to bridge this gap. By repurposing the pre-aligned knowledge of existing Image LLMs, RED-VILLM achieves state-of-the-art video understanding with significantly less data and fewer trainable parameters.

The Efficiency Gap in Video LLMs

Developing Video LLMs typically follows a two-stage paradigm: extensive pre-training on massive video-text datasets for modality alignment, followed by instruction fine-tuning. This process is computationally expensive and often redundant. Since a video is essentially a sequence of frames, the "modal gap" between an image and a video is much smaller than the gap between vision and text.

Existing Image LLMs like LLaVA or Qwen-VL already possess robust vision-text alignment. RED-VILLM argues that training Video LLMs from scratch ignores this foundational knowledge. Instead of a "roundabout" training process, the authors propose a "shortcut" that plugs temporal understanding directly into pre-aligned image architectures.

Comparison of training paradigms: (a) Image LLM, (b) Traditional Video LLM, and (c) the proposed RED-VILLM pipeline.

Methodology: The RED-VILLM Pipeline

The RED-VILLM architecture is built on three core pillars designed for efficiency and performance:

1. Spatio-Temporal Feature Extraction

To handle video data without overwhelming the LLM, RED-VILLM employs a dual-pooling strategy.

  • Temporal Pooling: This averages embeddings across the time dimension to capture global spatial features (the "what" of the scene).
  • Spatial Pooling: This averages across image dimensions to capture how the scene evolves over time (the "how" of the action). This combination ensures the model maintains a high-fidelity understanding of both static objects and dynamic movements.

2. The Plug-and-Play Temporal Module

The temporal module is the engine of the pipeline. Unlike traditional methods that initialize new modules randomly, RED-VILLM uses a "warm start" approach. The module is initialized with the weights of the pre-trained Image LLM’s alignment module. This ensures that the new temporal component inherits the existing visual-textual logic, drastically reducing the training time needed for convergence.

Detailed architecture of RED-VILLM featuring dual-pooling and the temporal module.

3. One-Stage Fine-Tuning

Because the base model is already well-aligned for visual concepts, RED-VILLM skips the massive pre-training phase entirely. It proceeds directly to instruction fine-tuning using a relatively small dataset (approx. 100k pairs). During this stage, the LLM backbone and vision encoder remain frozen; only the lightweight alignment and temporal modules are updated.

Experimental Results

The efficiency of RED-VILLM does not come at the cost of performance. In fact, it sets new benchmarks for resource-constrained video understanding.

  • Resource Efficiency: When based on LLaVA, RED-VILLM utilizes only 0.13B trainable parameters, a fraction of the 7B parameters often tuned in competing models like Video-LLaVA. Furthermore, it requires only 5% of the training data used by traditional pipelines.
  • Generative Excellence: On the ActivityNet generative benchmark, RED-VILLM outperformed Video-ChatGPT across metrics including correctness, detail, and temporal consistency.
  • Zero-Shot Performance: The model demonstrated superior generalization. On the MSVD-QA dataset, the Deepseek-VL-7B version of RED-VILLM achieved 73.5% accuracy, outperforming Video-ChatGPT's 64.9%.
  • Multilingual Support: The researchers also released the first dedicated Video LLM for the Chinese-speaking community, showing that the pipeline's efficiency translates well across different linguistic backbones.

Conclusion and Future Work

RED-VILLM demonstrates that the path to powerful Video LLMs doesn't have to be paved with massive datasets and prohibitive compute costs. By leveraging the "image-to-video" shortcut, the pipeline makes high-performance multimodal AI more accessible.

While the current implementation uses a fixed sampling rate of 100 frames—which may be a limitation for extremely long-form content—the success of the dual-pooling and weight-initialization strategies provides a clear roadmap for future developments in efficient multimodal learning.

Summary updated

Abstract

Covering from Image LLMs to the more complex Video LLMs, the Multimodal Large Language Models (MLLMs) have demonstrated profound capabilities in comprehending cross-modal information as numerous studies have illustrated. Previous methods delve into designing comprehensive Video LLMs through integrating video foundation models with primitive LLMs. Despite its effectiveness, such paradigm renders Video LLM's structure verbose and typically requires substantial video data for pre-training. Crucially, it neglects leveraging the foundational contributions of ready-made Image LLMs. In this paper, we introduce RED-VILLM, a Resource-Efficient Development pipeline which builds robust Video LLMs through leveraging the prior knowledge of Image LLMs. Specifically, since a video is naturally a combination of images along the temporal dimension, we devise a temporal adaptation plug-and-play structure, endowing the backbone Image LLM with the capability to grasp temporal information. Moreover, through applying this pipeline, we achieve the first Video LLM within the Chinese-speaking community. Extensive experiments demonstrate that Video LLMs developed through our approach surpass conventional Video LLMs, requiring minimal instructional data and training resources. Our approach highlights the potential for a more cost-effective and scalable advancement in multimodal models.
BibTeX
@misc{huang2024from,
  author = {Suyuan Huang and Haoxin Zhang and Yan Gao and Yao Hu and Zengchang Qin},
  title = {{From Image to Video, what do we need in multimodal LLMs?}},
  year = {2024},
  eprint = {2404.11865},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.11865},
}
Figure