In most computer vision tasks, "ground truth" is treated as an objective fact. A pixel either belongs to a car or it does not. However, in high-level video understanding—tasks like summarization, captioning, and highlight detection—this objective certainty disappears. If you ask ten people to summarize a ten-minute video, you will likely receive ten different sets of "key" frames and ten different descriptions.
This inherent subjectivity is not noise to be filtered out; it is a fundamental property of how humans perceive narrative and importance. Recent research suggests that failing to account for this variance leads to a "crisis of evaluation" and limits the ability of models to generalize to human preferences.
The Myth of the Gold Standard
The traditional approach to supervised learning relies on a single "gold standard" label. In video summarization, this usually means a set of frames selected by an annotator. However, as Aditya Khosla et al. note in Large-Scale Video Summarization Using Web-Image Priors, different viewers prioritize different aspects of a scene, making it nearly impossible to establish a stable benchmark.
This disagreement persists even at the most basic levels of video processing. Ralph Ewerth et al. found that even shot boundary detection—identifying where one camera cut ends and another begins—is hampered by subjectivity, as humans disagree on whether complex visual transitions constitute a "true" shot change. When the task moves to semantics, such as identifying "unusual" events, René Schuster et al. argue that the lack of objective benchmarks fundamentally limits quantitative evaluation.
The Evaluation Crisis
Because human annotators rarely agree, models trained on a single person's labels often fail to capture the broader consensus of what makes a video interesting. This creates a significant problem for evaluation metrics like the F1-score, which measures how closely a model's summary matches a ground-truth summary.
Mayu Otani et al. demonstrated the severity of this issue in Rethinking the Evaluation of Video Summaries, showing that current benchmarks are so flawed that random summaries can sometimes outperform state-of-the-art models. They argue that standard metrics often reward the underlying video segmentation structure rather than the actual selection of meaningful content. Similarly, Yang Wang argues that rigid metrics fail to account for the fact that multiple, non-overlapping summaries can be equally valid.
In the realm of language, Sheng Fang et al. observe that human-written captions vary significantly because different observers prioritize different actions or objects. A single ground-truth caption is an "incomplete query" that fails to capture the full semantic richness of the visual data.
Modeling the Distribution of Importance
To address this, researchers are moving away from deterministic models that seek a single "correct" answer and toward frameworks that embrace uncertainty and diversity.
- Stochastic Modeling: Instead of predicting a fixed summary, the authors of From Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning use latent stochastic variables to model the natural uncertainty in human interpretation. This allows the model to generate a variety of valid descriptions for the same video.
- Consensus-Based Benchmarks: Rather than picking one "best" annotation, newer datasets like FIOVA, introduced by Xuchen Li, provide multiple independent human descriptions for every video. This allows for metrics that weight events based on human consensus, rewarding models that align with the "cognitive richness" of human observers.
- Unsupervised and Weakly Supervised Learning: To bypass the "bottleneck" of subjective manual labels described by Yen-Ting Liu et al., many researchers are turning to unsupervised methods. For example, Kaiyang Zhou et al. use reinforcement learning to optimize for "diversity" and "representativeness"—mathematical proxies for importance that do not rely on biased human labels.
By treating subjectivity as a feature rather than a bug, the field is shifting toward systems that don't just "solve" a video, but rather understand the multiple valid ways a human might perceive it.
Key Foundational Work
University of Marburg at TRECVID 2006: Shot Boundary Detection and Rushes Task Results (2006) — contributed 3 concept elements to this direction.
A user attention model for video summarization (2002) — contributed 1 concept elements to this direction.