When evaluating how well an AI system understands, summarizes, or generates video, our automated metrics often look at the wrong things. Historically, the field has relied on evaluation metrics borrowed from machine translation—such as BLEU, ROUGE, and CIDEr—or from classical image processing, such as PSNR and SSIM.
These metrics share a fundamental flaw: they measure surface-level, token-for-token or pixel-for-pixel overlap rather than underlying semantic meaning. This creates a severe misalignment with human judgment, often referred to as the semantic evaluation gap.
The Illusion of Surface-Level Matching
To understand why traditional metrics fail, consider how a human describes a video versus how an n-gram metric evaluates it. If a video shows a golden retriever catching a disc, a model might output: "A canine runs after a flying toy on the grass." A human reference caption might read: "A dog chases a frisbee in the park."
Smeaton et al. in [TRECVID 2016: Evaluating Video Search, Video Event Detection, Localization, and Hyperlinking (TRECVID 2016)] observed that standard machine translation metrics prioritize exact word matching over the essence of the description, unfairly penalizing factually correct descriptions that use alternative phrasing. As Athar et al. demonstrate in [ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation], these mathematical overlap formulas exhibit remarkably low correlation with actual human judgment.
This problem compounds drastically as descriptions grow longer and more detailed. When a model generates a dense, paragraph-length summary of a video, the exact sequence of words matters far less than the overarching narrative flow. As Chai argues in [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark] and Liu et al. show in [DeVAn: Dense Video Annotation for Video-Language Models], traditional n-gram metrics like CIDEr actively penalize descriptive accuracy in long-form video summaries because they reward stylistic mimicry over semantic substance.
[Ground Truth]: "A dog chases a frisbee in the park."
[Model Output]: "A canine runs after a flying toy on the grass."
Lexical Overlap Metrics (BLEU/CIDEr): ❌ Low Score (Zero exact n-gram matches)
Semantic Evaluation Models: ✅ High Score (Identical underlying meaning)
A symmetrical failure occurs in video generation and prediction. If a model predicts a future sequence where an object is rendered with perfect, sharp fidelity but is positioned slightly to the left of the ground truth, pixel-level metrics like PSNR or SSIM will assign it a terrible score. Conversely, a model that outputs a blurry, non-committal average of all possible futures is often rewarded by these same metrics. Saxena et al. point out this exact pathology in [Clockwork Variational Autoencoders], demonstrating that pixel-wise alignment is a poor proxy for structural and semantic accuracy over long time horizons.
Why Surface Metrics Fail in Critical Contexts
The limitations of overlap metrics become dangerous when video understanding models are deployed in complex, safety-critical, or highly dynamic environments.
- Causal Dynamics and Actors: In traffic safety analytics, Wang et al. note in [The 8th AI City Challenge] that standard NLP metrics cannot verify whether a model has correctly identified the specific actors or causal dynamics involved in an accident. A high BLEU score might mask a catastrophic error, such as swapping the roles of the victim and the vehicle.
- Physical and Logical Constraints: For text-to-video generation, Mou et al. argue in [GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning] that pixel-level quality or basic text-image similarity scores fail to capture high-level attributes like physical logic, safety, and narrative creativity. Similarly, Chen et al. show in [VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generation] that word overlap cannot evaluate critical dimensions like motion trajectories or lighting conditions.
- Temporal Coherence: In video summarization, Zehni highlights in [DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization] that pixel-level similarity metrics are too brittle to distinguish between a high-quality, cohesive summary and a random selection of frames due to variations in camera angles and lighting. As argued by Zhang et al. in [Retrospective Encoders for Video Summarization] and Sharghi et al. in [Improving Sequential Determinantal Point Processes for Supervised Video Summarization], frame-matching metrics treat video segments as independent units, failing to assess whether the holistic meaning of the source video is preserved.
- Domain-Specific Correctness: In linguistically driven video synthesis, such as speech generation, Mathur demonstrates in [LIFI: Towards Linguistically Informed Frame Interpolation] that conventional vision metrics prioritize overall visual smoothness over phonetic correctness, scoring a video highly even if the synthesized lip movements fail to accurately represent the spoken words.
The Shift to Semantic and Model-Based Evaluation
To bridge this gap, researchers are developing evaluation methodologies that prioritize semantic fidelity, logical reasoning, and factual correctness over surface-level alignment.
Multimodal LLMs as Automated Judges
Instead of relying on rigid mathematical formulas, modern frameworks leverage compact, fine-tuned Multimodal Large Language Models (MLLMs) to act as automated judges. Waheed et al. demonstrate this in [VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding], where an automated judge model generates instance-specific rubrics and reasoning traces to evaluate video responses based on visual grounding rather than word overlap.
To handle open-ended video question answering, Chen et al. introduce in [AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering] an adversarial annotation process that allows an LLM judge to evaluate answers based on semantic correctness, achieving near-human alignment even when the model's phrasing differs entirely from the reference text.
Narrative-Aware and Temporally Indispensable Benchmarks
Another facet of this shift involves designing benchmarks that cannot be cheated by superficial language shortcuts or single-frame memorization. Chen et al. implement this in [MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding] by focusing on "temporally indispensable" questions that require holistic, multi-shot reasoning across time.
Similarly, Li et al. in [DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM] and Chen et al. in [PWS-DVC: Enhancing Weakly Supervised Dense Video Captioning With Pretraining Approach] advocate for narrative-aware evaluation frameworks that track the semantic evolution of a video over time, ensuring that the model understands the continuous flow of events.
Disentangling Language Style from Factual Truth
To completely isolate a model's visual understanding from its linguistic writing style, some researchers propose bypassing natural language evaluation altogether. Mahon et al. introduce this paradigm in [Knowledge Graph Extraction from Videos], mapping video content directly into structured, machine-readable Knowledge Graphs (KGs).
By representing video events as formal, logical facts (e.g., action(person, object)), the evaluation process can directly measure factual correctness and logical inference, eliminating the confounding factor of linguistic variation.
[Raw Video] ──> [Model Prediction] ──> Extract Knowledge Graph ──┐
├──> Exact Fact Matching
[Raw Video] ──> [Ground Truth] ──> Extract Knowledge Graph ──┘
By transitioning from surface-level lexical and pixel matching to model-based judges, narrative tracking, and structured knowledge extraction, the field is establishing evaluation protocols that truly measure whether an AI system understands the visual, causal, and logical reality of the video content.
Key Foundational Work
TRECVID 2016: Evaluating Video Search, Video Event Detection, Localization, and Hyperlinking (2016) — contributed 2 concept elements to this direction.
A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering (2017) — contributed 1 concept elements to this direction.
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016) — contributed 1 concept elements to this direction.