Performance over Random: A Robust Evaluation Protocol for Video Summarization Methods

Evlampios Apostolidis, Eleni Adamantidou, Alexandros I. Metsai, Vasileios Mezaris, Ioannis Patras

TL;DR

This paper identifies critical flaws in the standard evaluation of video summarization methods, showing that current metrics are highly sensitive to the specific data splits used, leading to unreliable comparisons between algorithms. To address this, they propose a new "Performance over Random" (PoR) metric that accounts for the difficulty of each data split, offering a more robust and representative evaluation.

What Problem Does This Paper Address?

The proliferation of video content online has fueled the development of automatic video summarization techniques, aiming to provide concise overviews of longer videos. However, evaluating these methods has relied on a standardized protocol using datasets like SumMe and TVSum. This protocol typically involves splitting data into training and testing sets and measuring performance using F-Score against human-generated summaries. The authors argue that this approach suffers from a significant problem: the "difficulty" of these randomly generated data splits varies wildly, leading to inconsistent and unreliable performance metrics. This makes it difficult to fairly compare different summarization algorithms, especially when they are evaluated on different sets of data splits.

Methods

The core of this paper is a critique of the existing evaluation protocol and the introduction of a new one.

  • Critique of Existing Protocol: The authors conducted a large-scale experiment using five publicly available video summarization algorithms and 50 randomly generated data splits for both SumMe and TVSum datasets. They compared these results to those reported in the original papers, finding significant discrepancies. They also observed that the performance curves of different algorithms across these splits showed remarkable similarity, suggesting that the splits themselves, rather than the algorithms' inherent capabilities, were driving the results. This led them to conclude that the F-Score, when applied to varying data splits, is not a reliable measure for comparing algorithms.
  • Proposed "Performance over Random" (PoR) Protocol: To overcome the limitations of the F-Score and its sensitivity to data splits, the authors propose the "Performance over Random" (PoR) metric. This metric is calculated as:

    $PoR = \frac{S}{R} \times 100$

    where $S$ is the performance score of the summarization method (e.g., F-Score) on a given data split, and $R$ is the performance score of a randomly generated summarizer on the same data split. A PoR score of 100 indicates performance equivalent to random chance, while scores above 100 suggest better-than-random performance. This approach inherently accounts for the difficulty of a data split: a higher random score on a difficult split means a method needs to perform exceptionally well to achieve a high PoR. They also explored a "Performance over Human" (PoH) metric, but found PoR to be more robust.

Datasets

The paper utilizes two well-established datasets for video summarization:

  • SumMe: Contains 25 videos with human annotations indicating selected video fragments for summaries.
  • TVSum: Features 50 videos across various genres, with frame-level importance scores obtained through crowd-sourcing.

The authors did not introduce a new dataset but rather focused on a more rigorous evaluation methodology using existing ones.

Key Insights and Ideas

  • Data Split Difficulty is a Major Confounding Factor: The most significant insight is that the random splits used in current evaluation protocols are not created equal. Some splits are inherently much harder than others, leading to a situation where an algorithm might perform well on one set of splits but poorly on another, making direct comparisons unreliable.
  • Random Performance as a Baseline: Using a random summarizer as a baseline is a powerful idea. It provides a dynamic, split-specific benchmark that reflects the inherent difficulty of the data. This is more informative than a fixed metric like F-Score alone.
  • PoR Offers Enhanced Robustness and Reliability: The PoR metric demonstrably reduces the variability of evaluation results across different data splits. This makes it a more reliable indicator of an algorithm's true performance and allows for fairer comparisons between methods, even if they were evaluated on different sets of data splits.
  • Similarity in Performance Curves: The observation that different algorithms' performance curves across splits look very similar is a strong indicator that the splits themselves are the primary driver of performance variation, not necessarily the algorithms' unique strengths.

Key Results

  • Unreliability of F-Score: Experiments showed that using only 5 data splits (common in many papers) led to significantly different rankings of algorithms compared to using 50 splits. The reported F-Scores from papers were often not congruent with performance on a larger set of splits.
  • PoR's Superior Representativeness: The Relative Standard Deviation (RSD) of performance metrics was significantly lower for PoR compared to F-Score (F1) and PoH across various evaluation settings. This indicates that PoR provides more consistent and representative performance estimates.
  • Improved Reliability of Comparisons: When comparing algorithms evaluated on different split-sets (simulating real-world scenarios where papers use different splits), the PoR metric showed a much smaller increase in ranking deviation compared to F-Score. This highlights PoR's ability to facilitate more reliable cross-paper comparisons. For instance, in mixed split-sets, the average ranking deviation for F-Score increased by up to 133% in TVSum, while for PoR, it increased by only 36%.

Limitations and Future Work

  • Computational Cost: The proposed PoR evaluation requires running a random summarizer for each data split, which adds computational overhead compared to simply calculating F-Scores.
  • Dependence on Random Summarizer Implementation: The effectiveness of PoR relies on a well-defined and consistent implementation of the random summarizer.
  • Future Work: The authors suggest that their PoR protocol can be used to re-evaluate existing video summarization methods and to guide future research towards developing more robust evaluation practices. They also make their code publicly available to encourage wider adoption.

Summary updated

Abstract

This paper proposes a new evaluation approach for video summarization algorithms. We start by studying the currently established evaluation protocol; this protocol, defined over the ground-truth annotations of the SumMe and TVSum datasets, quantifies the agreement between the user-defined and the automatically-created summaries with F-Score, and reports the average performance on a few different training/testing splits of the used dataset. We evaluate five publicly-available summarization algorithms under a large-scale experimental setting with 50 randomly-created data splits. We show that the results reported in the papers are not always congruent with their performance on the large-scale experiment, and that the F-Score cannot be used for comparing algorithms evaluated on different splits. We also show that the above shortcomings of the established evaluation protocol are due to the significantly varying levels of difficulty among the utilized splits, that affect the outcomes of the evaluations. Further analysis of these findings indicates a noticeable performance correlation among all algorithms and a random summarizer. To mitigate these shortcomings we propose an evaluation protocol that makes estimates about the difficulty of each used data split and utilizes this information during the evaluation process. Experiments involving different evaluation settings demonstrate the increased representativeness of performance results when using the proposed evaluation approach, and the increased reliability of comparisons when the examined methods have been evaluated on different data splits.
BibTeX
@inproceedings{apostolidis2020performance,
  author = {Evlampios Apostolidis and Eleni Adamantidou and Alexandros I. Metsai and Vasileios Mezaris and Ioannis Patras},
  title = {{Performance over Random: A Robust Evaluation Protocol for Video Summarization Methods}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413632},
}
Figure