STAT: Towards Generalizable Temporal Action Localization

Yangcen Liu, Ziyi Liu, Yuanhao Zhai, Wen Li, David S. Doermann, Junsong YuanarXiv 2024 · DOI · arXiv

Abstract

Weakly-supervised temporal action localization (WTAL) models often struggle with generalization, failing when applied to datasets with different temporal distributions. This paper identifies scale variance as a primary driver of this performance collapse and introduces STAT (Self-supervised Temporal Adaptive Teacher). STAT is a framework designed to refine localization boundaries and align temporal scales across distributions, significantly improving cross-dataset performance.

The Generalization Gap in WTAL

Weakly-supervised Temporal Action Localization (WTAL) involves identifying the start, end, and category of actions in videos using only video-level labels during training. While current models perform well under the "IID assumption"—where training and testing data share the same distribution—they often fail in real-world scenarios. For instance, a model trained on short, edited clips (THUMOS14) typically performs poorly on long, unscripted YouTube videos (ActivityNet).

The authors identify scale variance as the root cause. While models remain proficient at classifying actions in new environments, their localization accuracy collapses because the duration of the same action can vary drastically between datasets.

Action duration distributions across THUMOS14, HACS, and ActivityNet v1.2, illustrating the significant scale variance between datasets.

The STAT Framework

To address this, the authors propose the Generalizable Temporal Action Localization (GTAL) task and the STAT (Self-supervised Temporal Adaptive Teacher) framework. STAT employs a Mean Teacher architecture where a student model learns from a teacher model (a moving average of the student's weights).

Core Components:

  1. Temporal Refinement Module: This module mitigates scale variance using "Salience Sampling." It examines the context around video snippets; if a snippet has ambiguous attention scores, the module leverages high-confidence neighbors to refine boundaries, adapting the model to the target dataset's temporal scale.
  2. Alignment Module: STAT ensures effective learning through three primary losses:
  3. $L_{att}$ (Attention Alignment): Minimizes the difference between student and refined teacher attention scores.
  4. $L_{cas}$ (Classification Alignment): Ensures predicted action categories match using KL-divergence.
  5. $L_{cal}$ (Calibration Loss): A novel loss forcing consistency between low attention scores and background classification.

Datasets and Evaluation

The study utilizes three benchmarks: THUMOS14 (median duration ~3s), ActivityNet 1.2 (median duration ~28.5s), and HACS (median duration ~11.2s). Generalization is tested in a Cross-Distribution (CrD) setting, training on one dataset and testing on another using shared action classes.

Key Insights

  • Localization as the Bottleneck: Analysis shows that while video-level classification accuracy drops only slightly (~2.9%) in new environments, localization performance (mAP) suffers double-digit declines.
  • The "Ambiguous Middle": Errors predominantly occur at action boundaries where attention scores are neither clearly high nor low. STAT specifically targets these "ambiguous middle" snippets for refinement.
  • Scale as Domain Shift: Unlike traditional computer vision where domain shift involves lighting or texture, temporal localization is most affected by shifts in action duration.

Key Results

STAT demonstrates substantial improvements over state-of-the-art baselines like CO2-Net and DELU in cross-distribution scenarios.

Evaluation on ActivityNet 1.2 showing the massive mAP drop for baseline models when moving from same-distribution (SmD) to cross-distribution (CrD) settings.
  • ActivityNet to THUMOS14: STAT improved the average mAP from 23.4% (DELU baseline) to 53.6%, nearing the same-distribution performance of 58.2%.
  • THUMOS14 to ActivityNet: STAT boosted performance from 3.9% to 15.4% mAP.
  • Robustness: STAT consistently outperformed general domain generalization techniques like PGD and IRM, proving that temporal-specific refinement is superior for this task.

Limitations and Future Work

Current scale adaptation is class-agnostic, though different actions have unique characteristic scales. Future research could explore class-aware adaptation and methods to account for varying human annotation styles across datasets. Additionally, reducing the initial training cost by leveraging unlabeled data in the target distribution remains an open challenge.

Summary updated

Abstract

Weakly-supervised temporal action localization (WTAL) aims to recognize and localize action instances with only video-level labels. Despite the significant progress, existing methods suffer from severe performance degradation when transferring to different distributions and thus may hardly adapt to real-world scenarios . To address this problem, we propose the Generalizable Temporal Action Localization task (GTAL), which focuses on improving the generalizability of action localization methods. We observed that the performance decline can be primarily attributed to the lack of generalizability to different action scales. To address this problem, we propose STAT (Self-supervised Temporal Adaptive Teacher), which leverages a teacher-student structure for iterative refinement. Our STAT features a refinement module and an alignment module. The former iteratively refines the model's output by leveraging contextual information and helps adapt to the target scale. The latter improves the refinement process by promoting a consensus between student and teacher models. We conduct extensive experiments on three datasets, THUMOS14, ActivityNet1.2, and HACS, and the results show that our method significantly improves the Baseline methods under the cross-distribution evaluation setting, even approaching the same-distribution evaluation performance.
BibTeX
@misc{liu2024stat,
  author = {Yangcen Liu and Ziyi Liu and Yuanhao Zhai and Wen Li and David S. Doermann and Junsong Yuan},
  title = {{STAT: Towards Generalizable Temporal Action Localization}},
  year = {2024},
  eprint = {2404.13311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.13311},
}
Figure