VideoAds for Fast-Paced Video Understanding: Where Opensource Foundation Models Beat GPT-4o & Gemini-1.5 Pro

Zheyuan Zhang, Monica Dou, Linkai Peng, Hongyi Pan, Ulas Bagci, Boqing Gong

Introduction

Modern Multi-modal Large Language Models (MLLMs) have shown remarkable progress in general video understanding. However, most existing benchmarks evaluate models on long-form content or simple action recognition tasks. These "lazy" sampling approaches often fail when confronted with information-dense, fast-paced videos like advertisements. Advertisement videos typically feature rapid scene transitions, complex narrative structures, and subtle persuasive cues condensed into 60–120 seconds. To address this gap, researchers have introduced VideoAds, a benchmark specifically designed to challenge MLLMs in high-frequency visual and temporal reasoning.

The Complexity Gap in Video Benchmarking

A core issue identified in current research is the lack of a formal metric for video difficulty. This paper proposes Video Complexity ($V_{cpx}$), a quantitative measure derived from the variance of visual features (extracted via DINO-v2) over time. High $V_{cpx}$ scores correspond to frequent cuts and dynamic interactions, whereas low scores indicate static, slideshow-like content.

Figure 1: Comparison of VideoAds against existing benchmarks. VideoAds exhibits significantly higher video complexity compared to traditional datasets like VideoMME and EgoSchema.

As shown in Figure 1, VideoAds occupies a unique position in the benchmarking landscape, offering a level of visual density that far exceeds established datasets like VideoMME, EgoSchema, and MVBench. The study proves that video length is a poor proxy for difficulty; a 30-second ad can be more computationally challenging than a 10-minute vlog if it contains more "semantic shifts" per second.

Methodology: Task Triangulation and Human Annotation

The VideoAds evaluation framework moves beyond simple identification by categorizing questions into three hierarchical tiers:

  1. Visual Finding: Identification of concrete objects or patterns (e.g., "What color is the car?").
  2. Visual Summary: Synthesis of thematic elements (e.g., "What is the primary product?").
  3. Visual Reasoning: High-order inference regarding narrative intent and causality (e.g., "Why did the character make that choice?").
Figure 3: Examples of the three-tiered evaluation tasks in VideoAds, including reasoning steps required for high-order visual reasoning questions.

To ensure the benchmark cannot be bypassed through linguistic shortcuts or "common sense" guessing, the researchers employed human experts to curate 200 high-quality videos and 1,100 manually annotated multiple-choice pairs. This human-in-the-loop approach ensures that distractors are genuinely confusing and require actual visual processing to resolve, as evidenced by a text-only baseline (GPT-4o) achieving only ~21% accuracy.

Key Results and the Rise of Open-Source

The evaluation of leading MLLMs on VideoAds yielded several surprising insights:

  • Top Performance: Gemini 2.5 Pro led the pack with 80.04% accuracy.
  • Open-Source Superiority: In a significant shift, the open-source Qwen2.5-VL-72B outperformed major proprietary models, scoring 73.35% compared to GPT-4o (66.82%) and Gemini-1.5 Pro (69.66%). This suggests that specialized temporal reasoning capabilities are rapidly maturing in the open-source community.
  • The Reasoning Bottleneck: While models performed competently at "Visual Finding," there was a sharp decline in performance for "Visual Reasoning" tasks. Human experts achieved 94.27%, highlighting a persistent gap in "narrative intuition" that models still lack.
  • Multimodal Synergy: The inclusion of speech transcripts consistently improved performance, indicating that current models rely heavily on textual cues to supplement their visual understanding of complex narratives.

Conclusion

VideoAds demonstrates that semantic density and rapid editing present a much greater challenge for current MLLMs than simple duration. While proprietary models like Gemini 2.5 Pro lead, the strong performance of Qwen2.5-VL-72B marks a turning point for open-source video foundation models. Future developments must focus on high-FPS architectures capable of processing dense visual information without prohibitive computational costs.

Summary updated

BibTeX
@inproceedings{zhang2025videoads,
  author = {Zheyuan Zhang and Monica Dou and Linkai Peng and Hongyi Pan and Ulas Bagci and Boqing Gong},
  title = {{VideoAds for Fast-Paced Video Understanding: Where Opensource Foundation Models Beat GPT-4o \& Gemini-1.5 Pro}},
  booktitle = {ICCV},
  year = {2025},
  url = {https://openreview.net/forum?id=AI1HUR1Lu8},
}
Figure