Abstract
Weakly-supervised temporal action localization (WTAL) models often struggle with generalization, failing when applied to datasets with different temporal distributions. This paper identifies scale variance as a primary driver of this performance collapse and introduces STAT (Self-supervised Temporal Adaptive Teacher). STAT is a framework designed to refine localization boundaries and align temporal scales across distributions, significantly improving cross-dataset performance.
The Generalization Gap in WTAL
Weakly-supervised Temporal Action Localization (WTAL) involves identifying the start, end, and category of actions in videos using only video-level labels during training. While current models perform well under the "IID assumption"—where training and testing data share the same distribution—they often fail in real-world scenarios. For instance, a model trained on short, edited clips (THUMOS14) typically performs poorly on long, unscripted YouTube videos (ActivityNet).
The authors identify scale variance as the root cause. While models remain proficient at classifying actions in new environments, their localization accuracy collapses because the duration of the same action can vary drastically between datasets.
The STAT Framework
To address this, the authors propose the Generalizable Temporal Action Localization (GTAL) task and the STAT (Self-supervised Temporal Adaptive Teacher) framework. STAT employs a Mean Teacher architecture where a student model learns from a teacher model (a moving average of the student's weights).
Core Components:
- Temporal Refinement Module: This module mitigates scale variance using "Salience Sampling." It examines the context around video snippets; if a snippet has ambiguous attention scores, the module leverages high-confidence neighbors to refine boundaries, adapting the model to the target dataset's temporal scale.
- Alignment Module: STAT ensures effective learning through three primary losses:
- $L_{att}$ (Attention Alignment): Minimizes the difference between student and refined teacher attention scores.
- $L_{cas}$ (Classification Alignment): Ensures predicted action categories match using KL-divergence.
- $L_{cal}$ (Calibration Loss): A novel loss forcing consistency between low attention scores and background classification.
Datasets and Evaluation
The study utilizes three benchmarks: THUMOS14 (median duration ~3s), ActivityNet 1.2 (median duration ~28.5s), and HACS (median duration ~11.2s). Generalization is tested in a Cross-Distribution (CrD) setting, training on one dataset and testing on another using shared action classes.
Key Insights
- Localization as the Bottleneck: Analysis shows that while video-level classification accuracy drops only slightly (~2.9%) in new environments, localization performance (mAP) suffers double-digit declines.
- The "Ambiguous Middle": Errors predominantly occur at action boundaries where attention scores are neither clearly high nor low. STAT specifically targets these "ambiguous middle" snippets for refinement.
- Scale as Domain Shift: Unlike traditional computer vision where domain shift involves lighting or texture, temporal localization is most affected by shifts in action duration.
Key Results
STAT demonstrates substantial improvements over state-of-the-art baselines like CO2-Net and DELU in cross-distribution scenarios.
- ActivityNet to THUMOS14: STAT improved the average mAP from 23.4% (DELU baseline) to 53.6%, nearing the same-distribution performance of 58.2%.
- THUMOS14 to ActivityNet: STAT boosted performance from 3.9% to 15.4% mAP.
- Robustness: STAT consistently outperformed general domain generalization techniques like PGD and IRM, proving that temporal-specific refinement is superior for this task.
Limitations and Future Work
Current scale adaptation is class-agnostic, though different actions have unique characteristic scales. Future research could explore class-aware adaptation and methods to account for varying human annotation styles across datasets. Additionally, reducing the initial training cost by leveraging unlabeled data in the target distribution remains an open challenge.
Summary updated