TL;DR
This paper introduces VideoAds, a novel benchmark dataset designed to evaluate multi-modal large language models (MLLMs) on their ability to understand fast-paced advertisement videos. These videos, characterized by rapid scene transitions and complex narratives, pose a significant challenge for current MLLMs, and VideoAds aims to bridge this gap by offering a more rigorous evaluation.
What Problem Does This Paper Address?
Understanding video content is a rapidly evolving area in AI, with Multi-Modal Large Language Models (MLLMs) showing great promise. However, existing benchmarks often focus on simpler video types like action recognition or general YouTube clips. Advertisement videos, while short, are densely packed with information, employing sophisticated storytelling, visual cues, and rapid scene changes to engage viewers. This complexity makes them particularly challenging for MLLMs, especially in tasks requiring temporal reasoning, causal inference, and understanding nuanced narratives. The authors argue that current benchmarks don't adequately capture these challenges, leading to an incomplete understanding of MLLMs' capabilities in real-world scenarios.
Methods
The core contribution is the VideoAds dataset, a collection of 200 high-quality advertisement videos. To create this dataset, the authors:
- Collected Videos: They gathered 2,700 advertisement videos from YouTube across various commercial domains.
- Filtered and Curated: A systematic filtering process removed irrelevant, low-quality, or short videos, resulting in the final 200 videos.
- Developed a Complexity Metric: They introduced a novel quantitative measure for "video complexity" based on the variance of visual features over time (using DINO-v2). This metric aims to capture how challenging a video is for MLLMs, particularly concerning temporal dynamics and rapid scene changes.
- Annotated VQA Tasks: Human experts manually created 1,100 question-answering (VQA) tasks across three categories:
- Visual Finding: Extracting concrete visual patterns (e.g., object recognition, attribute identification).
- Visual Summary: Summarizing key events and thematic elements.
- Visual Reasoning: Performing higher-order inference, causal analysis, and emotional interpretation.
- They also used LLava-Next-72B and GPT-4o to generate candidate incorrect answers, which were then refined into standardized four-option multiple-choice questions.
Datasets
- VideoAds: The newly introduced benchmark dataset, consisting of 200 advertisement videos with 1,100 manually annotated VQA tasks.
- Existing Benchmarks: The paper compares VideoAds to several existing video understanding benchmarks, including TGIF-QA, STAR, NEXT-QA, MSVD-QA, MSRVTT-QA, ActivityNet-QA, TempCompass, MVBench, Video-Bench, EgoSchema, AutoEval-Video, and Video-MME. The comparison highlights VideoAds' unique emphasis on high video complexity and temporal dynamics.
Key Insights and Ideas
- Advertisement Videos as a Unique Challenge: The paper strongly emphasizes that advertisement videos are not just "short videos" but represent a distinct category of visual content with inherent complexity that existing benchmarks fail to capture.
- Temporal Complexity is Key: The authors' proposed complexity metric highlights that rapid scene transitions and dense information, common in ads, are more challenging for MLLMs than simply longer video durations.
- MLLMs Struggle with Reasoning: A significant gap exists between MLLMs' performance on visual finding tasks and their performance on visual summary and, especially, visual reasoning tasks. This suggests current models are better at recognizing static elements than understanding dynamic narratives.
- Open-Source Models Show Promise: Surprisingly, the open-source Qwen2.5-VL-72B model outperformed proprietary models like GPT-4o and Gemini 1.5 Pro on the VideoAds benchmark, particularly in summarization and reasoning.
- Audio Transcripts are Crucial: Incorporating speech audio transcripts significantly boosts MLLM performance, indicating that current models might be overly reliant on visual information and underutilizing cross-modal reasoning.
- Chain-of-Thought (CoT) Benefits Vary: While CoT prompting generally helps, its effectiveness differs across models, with larger models sometimes showing a performance drop, suggesting that current CoT strategies might not be optimal for complex temporal reasoning.
Key Results
- Human Performance Benchmark: Human experts achieved a remarkable 94.27% accuracy on VideoAds, highlighting the significant gap between human understanding and current MLLMs.
- Top MLLM Performance: Gemini 2.5 Pro achieved the highest MLLM accuracy at 80.04%.
- Qwen2.5-VL-72B Outperforms Proprietary Models: The open-source Qwen2.5-VL-72B model achieved 73.35% accuracy, surpassing GPT-4o (66.82%) and Gemini 1.5 Pro (69.66%).
- Task Performance Disparity: MLLMs performed best on Visual Finding (e.g., GPT-4o at 75.06%), followed by Visual Summary, and struggled most with Visual Reasoning (e.g., GPT-4o at 62.26%).
- Audio Transcript Impact: Adding audio transcripts improved performance across all tested MLLMs, with gains particularly noticeable in reasoning tasks.
- Text-Only Baseline: A GPT-40 model prompted with text-only input achieved only 21.27% accuracy, confirming that VideoAds effectively eliminates text-based shortcuts.
Limitations and Future Work
- Dataset Size: While VideoAds offers high complexity, the dataset size of 200 videos is relatively small compared to some other benchmarks.
- Temporal Modeling: The paper explicitly calls for advancements in MLLMs' temporal modeling capabilities to better handle the rapid scene transitions and complex narratives found in advertisement videos.
- Cross-Modal Reasoning: The significant improvement with audio transcripts suggests that further research into robust cross-modal reasoning is needed.
- CoT Strategies: The varied impact of CoT prompting indicates a need for more sophisticated CoT strategies tailored for temporal and narrative understanding in videos.
- High FPS Sampling: The authors suggest that future research should explore models that can effectively process high FPS sampling for a more granular understanding of fast-paced videos.
Summary updated