An Empirical Study for Representations of Videos in Video Question Answering via MLLMs

Zhi Li, Yanan Wang, Hao Niu, Julio Vizcarra, Masato TayaarXiv 2025 · DOI · arXiv

Introduction

Video Question Answering (VideoQA) presents a significant challenge for Multimodal Large Language Models (MLLMs) due to the sheer volume of spatial, temporal, and auditory data inherent in video streams. While models like LLaVA and Qwen-VL have demonstrated impressive capabilities, the computational cost of processing every frame is often prohibitive. This paper provides a systematic "bang-for-your-buck" analysis, investigating which video representations—visual frames, audio, or textual subtitles—offer the best balance of accuracy and efficiency.

Methodology: Isolating Modalities

The study introduces a framework designed to isolate and combine four primary input types to determine their individual and collective impact on model performance.

The proposed framework consists of two stages: representation extraction (visual, audio, and textual) and video question answering using an MLLM.

The four input modalities evaluated are:

  1. Question Only (Q): Establishes a baseline for "blind" performance, relying solely on the model's internal knowledge.
  2. Subtitles (S): Textual transcripts generated via a pipeline of Voice Activity Detection (VAD) and Whisper-Large-v3.
  3. Visual Frames (V): Uniformly sampled frames processed through a visual encoder (e.g., CLIP) and an MLP projector.
  4. Audio Signals (A): Raw audio features extracted using specialized encoders, such as those in the Video-SALMONN architecture.

These combinations were tested across 7B and 72B parameter versions of LLaVA-video and Qwen2.5-vl to assess how model scale influences multimodal integration.

Experimental Benchmarks

The researchers utilized two rigorous benchmarks for long-context video understanding:

  • VideoMME: Comprising 900 YouTube videos (2 minutes to 1+ hours) across diverse domains like documentaries and movies.
  • LongVideoBench: Featuring 753 web videos (15 seconds to 1 hour) focused on complex, multi-scene reasoning tasks.

Key Findings

The empirical results reveal several critical insights into how MLLMs process video data:

1. The Power of Textual Subtitles

For long-form content, subtitles emerge as the "MVP" modality. Because MLLMs are fundamentally language-centric, they can process vast amounts of compressed text (subtitles) more effectively than a sparse set of sampled visual frames. Adding subtitles to a visual-only model (Q+V) provides a substantial accuracy boost with negligible increases in VRAM and latency.

2. The Visual Bottleneck

While visual frames are essential for the highest accuracy, they are the primary driver of resource consumption. Including visuals nearly doubled VRAM requirements for 7B models and significantly increased inference latency. For hour-long videos, processing visual tokens can extend inference times from minutes to hours.

3. Modality Comparison: Text vs. Audio

Interestingly, the study found that MLLMs perform better with textual transcripts (subtitles) than with raw audio features. On Qwen-2.5-vl models, the Subtitle + Visual combination outperformed the Audio + Visual combination by approximately 3.6% to 3.7%, suggesting that current architectures are better optimized for text-based reasoning.

Performance Results

The comprehensive evaluation across benchmarks confirms that the multi-modal approach yields the best results.

Detailed performance metrics for various model configurations and input combinations on the VideoMME and LongVideoBench benchmarks.

As shown in the results, the Question + Subtitles + Visuals (Q+S+V) configuration consistently achieved the highest scores. For instance, the Qwen2.5-vl-72B model reached an overall accuracy of 78.4% on VideoMME using this combination. Even the "Question Only" baseline for 72B models outperformed random guessing by ~20%, indicating that these models leverage significant world knowledge to infer answers from context.

Conclusion and Future Directions

This study highlights a critical trade-off in VideoQA: while visual data is paramount for detail, textual subtitles provide a highly efficient alternative for long-context understanding. Current MLLMs are primarily bottlenecked by token length limits, which necessitates sparse visual sampling and potentially leads to the loss of fine-grained details.

Future research should explore methods to bridge the gap between heavy pixel data and efficient text, such as "dense captioning" where models generate their own descriptive text for visual events. Additionally, fine-tuning models specifically for subtitle-heavy, long-video reasoning remains a promising avenue for improving both performance and efficiency in real-world applications.

Summary updated

BibTeX
@misc{li2025empirical,
  author = {Zhi Li and Yanan Wang and Hao Niu and Julio Vizcarra and Masato Taya},
  title = {{An Empirical Study for Representations of Videos in Video Question Answering via MLLMs}},
  year = {2025},
  eprint = {2510.12299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12299},
}
Figure