QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering

Woojun Jung, Junyeong KimEMNLP 2025

TL;DR

Evaluating video summaries is tough because current methods rely on human-written references, which are costly and don't capture nuanced aspects. This paper introduces QEVA, a novel reference-free metric that uses multimodal question answering to assess summaries against the source video across coverage, factuality, and chronology. QEVA shows significantly higher correlation with human judgments than existing metrics, offering a more scalable and practical solution.

What Problem Does This Paper Address?

The explosive growth of video content has led to significant advances in video-to-text summarization, with models generating increasingly comprehensive summaries. However, a major bottleneck in this field is the lack of reliable and practical evaluation methods. Traditional metrics like ROUGE and METEOR, which compare generated summaries to human-written references, are expensive to create and may not capture subtle semantic nuances. More recent LLM-based approaches also often depend on these references or can be prone to biases. This reliance on human references limits the scalability and practicality of evaluating video summarization systems, hindering progress. The paper aims to bridge this gap by proposing a fully reference-free evaluation metric.

Methods

QEVA evaluates video summaries by leveraging multimodal question answering (QA). The core idea is that a high-quality summary should effectively substitute for the original video content. QEVA breaks down summary quality into three key dimensions:

  • Coverage: Assesses if the summary comprehensively covers essential content, key events, and main messages of the source video.
  • Factuality: Verifies if the summary accurately reflects details from the source video without introducing hallucinations or unsupported information.
  • Chronology: Evaluates if the summary preserves the correct order of events from the source video, including handling non-linear structures.

QEVA employs a pipeline where a Video-Large Multimodal Model (Video-LMM) generates relevant questions based on the source video and candidate summary for each dimension. These questions are then answered by a Large Language Model (LLM) using only the generated summary. The accuracy of these answers is used to compute scores for each dimension. A two-stage automatic filtering process is used to remove trivial or low-quality QA pairs. The final QEVA score is the average of the scores for Coverage, Factuality, and Chronology.

Datasets

The paper introduces MLVU(VS)-Eval, a new benchmark dataset for evaluating video summarization metrics. This dataset is derived from the MLVU benchmark and comprises 800 candidate summaries generated by four widely-used Video-LMM models (GPT-40, InternVL3-8B, Qwen2.5-VL-7B, and Video-LLaVA-7B) for 200 video clips. These summaries were annotated by 20 human evaluators on a 5-point Likert scale for Coverage, Factuality, and Chronology, achieving a substantial inter-annotator agreement (Krippendorff's α = 0.68).

Key Insights and Ideas

  • Reference-Free is Key: The most significant insight is that a robust evaluation metric for video summarization can be achieved without relying on human-written reference summaries. This directly addresses the major bottleneck of cost and scalability.
  • Multimodal QA as the Solution: Using multimodal QA as the core mechanism allows for a deeper, more nuanced evaluation than simple text-based comparisons. It forces the system to "understand" the video content through the summary.
  • Decomposition into Dimensions: Breaking down summary quality into Coverage, Factuality, and Chronology provides a more interpretable and comprehensive evaluation framework, acknowledging that a good summary needs to excel in all these aspects.
  • Synergy of Video-LMMs and LLMs: The approach effectively combines the strengths of Video-LMMs for video understanding and question generation with LLMs for text-based question answering and evaluation.

Key Results

QEVA consistently demonstrated significantly higher correlations with human judgments compared to existing reference-based (e.g., BLEU, ROUGE, METEOR, BERTScore) and LLM-based evaluation methods (e.g., MLVU, Video-LMM Judge) on the MLVU(VS)-Eval benchmark. This was measured using Kendall's τb, τc, and Spearman's ρ. The paper also shows that QEVA maintains stable and positive correlations across different Video-LMM summarization models, highlighting its robustness and generalizability. Ablation studies confirmed that QEVA performs strongly across each individual evaluation criterion (Coverage, Factuality, Chronology).

Limitations and Future Work

  • Hallucination: QEVA's reliance on LMMs for question generation means there's a possibility of hallucinated content in questions or answers, though filtering mitigates this.
  • API Cost and Processing Speed: The use of powerful LLMs and Video-LMMs incurs significant computational costs and can limit real-time applications or large-scale evaluations.
  • Post-processing: Occasional deviations from expected output formats may require additional post-processing.
  • LLM Bias: QEVA might exhibit a slight bias favoring summaries produced by certain Video-LMMs, similar to other LLM-based evaluations.

Future work could involve further investigation into mitigating LLM biases and exploring more efficient model combinations to reduce computational costs. The authors also suggest that the developed benchmark and metric will facilitate future research in video summarization and evaluation methods.

Summary updated

BibTeX
@inproceedings{jung2025qeva,
  author = {Woojun Jung and Junyeong Kim},
  title = {{QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure