In image classification, a label applies uniformly to the entire asset: an image of a cat contains a cat. Video understanding, however, introduces a challenging dimension: time. When a ten-minute video is labeled "cooking," the actual action of cooking might only occur for a brief thirty-second window. The remaining nine and a half minutes might consist of an introduction, cleaning, or a static shot of a kitchen counter.
As noted by Yunjie Xie et al. and Ghadiyaram et al., applying a single global tag to an entire untrimmed video introduces severe temporal label noise. The target action or event frequently occupies only a small fraction of the total duration, creating a weak correlation between the video-level label and the majority of its constituent frames (Xing Zhang et al.).
The Mechanism of Failure: Why Global Labels Confuse Local Models
When standard deep learning models are trained on untrimmed videos, they typically distribute the global video label uniformly across all frames or short sub-clips. This practice creates a fundamental "false label" problem (Lan et al.).
Untrimmed Video Timeline:
[ Intro (Normal) ] -> [ Actual Event (Target) ] -> [ Outro (Normal) ]
| | |
False Positive True Positive False Positive
(Label Noise) (Label Noise)
During back-propagation, treating every frame as equally representative forces the feature extractor to learn task-specific patterns from completely irrelevant background data (Song et al.). Instead of isolating the unique features of the target action, the model inadvertently associates benign background information or preparatory movements with the target class, severely degrading its generalization performance.
The Sources of Temporal Noise
Temporal label noise stems from two primary paradigms in dataset creation:
- Weakly Supervised Automation: To scale video datasets to millions of samples, researchers often harvest web videos using automated metadata, scripts, or subtitles. However, as Sivic et al. and Prajwal et al. demonstrate, these textual cues are frequently misaligned with the actual visual events. This temporal offset creates a noisy training environment where text descriptions predict actions that have either already occurred or have not yet begun (Eyzaguirre et al.).
- Human Annotation Boundaries: Even when human annotators are deployed, defining the exact start and end of an action is highly subjective. The authors of Trespassing the Boundaries show that different individuals rarely agree on precise timestamps for object interactions. Furthermore, dense frame-by-frame manual annotation introduces high variance and contradictory signals due to human fatigue and boundary ambiguity (Yan et al.; Wuyang Li et al.).
Traditional label noise techniques developed for static images treat samples as independent and identically distributed (i.i.d.). Zixiao Wang et al. observe that these image-based approaches fail in video because they ignore the high dimensionality and sequential dependencies of video data. For instance, randomly sampling segments to reduce data redundancy can inadvertently alter the perceived meaning of an activity, transforming a complex multi-stage action into an ambiguous sub-action that mimics an entirely different class (Hanchao Liu et al.).
Core Mechanisms for Denoising Time
To train robust video models without pristine frame-level annotations, researchers have developed specialized architectures designed to isolate salient signals from temporal noise.
Multiple Instance Learning and Normality Suppression
A highly effective framing for weakly supervised video analysis—particularly in video anomaly detection—is Multiple Instance Learning (MIL). Under the MIL framework, an entire video is treated as a "bag," and individual short clips or frames are treated as "instances." If a video is labeled abnormal, the bag is marked positive, meaning it contains at least one abnormal instance, though the vast majority of frames are normal (Feng; Zaheer et al.).
To prevent the model from mistakenly flagging normal background frames as anomalies, Park et al. utilize "normality prototypes" learned from guaranteed noise-free videos. By explicitly modeling what standard background behavior looks like, the network can suppress the activations of normal segments within an anomalous video, effectively isolating the sparse, true positive signals.
Cross-Validation and Co-Teaching
When temporal boundaries are highly corrupted, models tend to overfit to the erroneous timestamps. To mitigate this, Daizong Liu et al. introduce a Co-Teaching Regularizer framework.
+-----------------+
| Noisy Labels |
+--------+--------+
|
+---------------+---------------+
| |
v v
+--------------------+ +--------------------+
| Model A | | Model B |
| Filters Low-Noise | | Filters Low-Noise |
| Samples | | Samples |
+----------+---------+ +----------+---------+
| |
+--------> Cross-Train <--------+
By training two distinct neural networks simultaneously, each model filters out high-loss (likely noisy) samples and passes the remaining clean samples to its partner network. This cross-validation prevents either network from treating corrupted temporal boundaries as absolute truth, stabilizing the learning process even under heavy label corruption.
Contrastive Denoising and Temporal Alignment
In semi-supervised and self-supervised settings, models often rely on generating pseudo-labels for unlabeled video segments. However, if these pseudo-labels are inaccurate, errors propagate rapidly through contrastive learning loops.
To break this error loop, Jingjing Li et al. propose Denoised Cross-video Contrast (DCC), which filters out unreliable pseudo-labels by comparing region-level features across a global memory bank. In parallel, Dave leverages temporal alignability via Dynamic Time Warping (DTW). By matching the phases of an unlabeled video against a known reference category, the system verifies whether the temporal progression matches the assigned class, ensuring that semantically similar but distinct action phases are not mislabeled.
Key Foundational Work
DFKI-IUPR participation in TRECVID'09 High-level Feature Extraction Task (2009) — contributed 2 concept elements to this direction.
NII, Japan at MediaEval 2012 Violent Scenes Detection Affect Task (2012) — contributed 1 concept elements to this direction.
Improved facial expression recognition via uni-hyperplane classification (2012) — contributed 1 concept elements to this direction.