Improving Weakly Supervised Temporal Action Localization by Exploiting Multi-resolution Information in Temporal Domain

Rui Su, Dong Xu, Luping Zhou, Wanli Ouyang

TL;DR

This paper introduces a novel two-stage framework for weakly supervised temporal action localization by enforcing consistency across different temporal resolutions. By leveraging the fact that video content remains semantically identical regardless of playback speed, the method generates and iteratively refines high-quality frame-level pseudo-labels, significantly boosting localization accuracy without requiring expensive frame-by-frame annotations.

What Problem Does This Paper Address?

Temporal Action Localization (TAL) aims to pinpoint the exact start and end times of specific actions within long, untrimmed videos. While fully supervised models excel at this, they require meticulous, frame-level boundary annotations, which are incredibly expensive and highly subjective to produce.

Weakly Supervised TAL (WSTAL) offers a cheaper alternative by training models using only video-level labels (e.g., knowing a video contains a "frisbee catch" without knowing when it happens). However, standard WSTAL methods rely on Class Activation Sequences (CAS) that tend to only activate during the most distinct, "discriminative" frames of an action (like the exact moment the frisbee is caught), completely missing the preparation or wind-down phases and leading to fragmented or incomplete localizations.

Methods

The authors tackle this issue by exploiting temporal multi-resolution information in a two-stage approach:

  1. Initial Label Generation (ILG) Module: Built on top of a two-stream (RGB and optical flow) backbone, this module feeds both original-resolution and temporally scaled (resampled) video features into the network. It introduces two self-supervised constraints to clean up the initial predictions:
  2. Temporal Multi-Resolution Consistency Loss: Forces the network to output identical action predictions for the same video content, regardless of its temporal scale (speed).
  3. Cross-Stream Consistency Loss: Aligns predictions between the appearance (RGB) and motion (optical flow) streams. High-confidence predictions from this module form the initial frame-level pseudo-labels.

  4. Progressive Temporal Label Refinement (PTLR) Framework: An iterative co-training framework consisting of two distinct networks that refine each other's pseudo-labels over multiple stages:

  5. Network-OTS (Original Temporal Scale): Generates predictions from the original video duration.
  6. Network-RTS (Reduced Temporal Scale): Uses a multi-branch structure with 1D convolutions to extract features at multiple reduced temporal resolutions, which are then upsampled and combined.
Fig. 1. The overall framework of our Initial Label Generation (ILG) module. Our framework consists of two networks: (a) Original-resolution Network, (b) Transformed-resolution Network. The temporal scale transform operation is applied on the RGB/flow feature FRGB/FFlow to obtain the transformed RGB/flow feature ˆFRGB/ˆFFlow. The networks in (a) and (b) take FRGB/FFlow and ˆFRGB/ˆFFlow as the input and use Bas-Net [7] to generate CASs ARGB/AFlow and ˆARGB/ ˆAFlow, respectively. and note each network also consists of a base branch and a suppression branch.
The Progressive Temporal Label Refinement (PTLR) framework

Datasets

The approach was validated on two standard video action benchmarks:

  • THUMOS14: Features 20 action classes. The authors used 200 validation videos for training (with video-level labels) and 212 test videos for evaluation.
  • ActivityNet v1.3: A much larger dataset containing nearly 20,000 videos across 200 activity classes. Models were trained on the training set and evaluated on the validation set.

Key Insights and Ideas

  • Temporal Invariance as a Supervisor: The core "aha moment" is that a video's meaning doesn't change if you speed it up or slow it down. Standard networks often yield wildly inconsistent frame-level predictions when a video is temporally scaled; forcing multi-resolution consistency acts as a powerful, annotation-free regularizer.
  • Co-Training Across Scales: Network-OTS and Network-RTS act as two different "views" of the same data. Exchanging pseudo-labels between an original-scale network and a multi-scale reduced network helps recover full action boundaries rather than just peak action moments.
  • Progressive Threshold Lowering: By gradually lowering the selection threshold across training stages, the model safely transitions from learning only from the easiest, most obvious frames to capturing subtle, less discriminative parts of the action sequence.

Key Results

  • THUMOS14: At an Intersection-over-Union (IoU) threshold of 0.5, the proposed method achieves a mean Average Precision (mAP) of 29.7%, outperforming the baseline by 2.7% and beating several older fully supervised methods.
  • ActivityNet v1.3: It achieves an average mAP of 23.7%, a 1.5% improvement over the baseline.
  • Qualitative Improvements: Visualizations demonstrate that while baseline methods split a single continuous action into fragmented segments because they miss less distinct frames, this multi-resolution approach successfully localizes the entire action duration smoothly.

Limitations and Future Work

The authors note an inherent trade-off in the progressive refinement stages: as more stages are added and the selection threshold is lowered to capture more frames, the precision of the pseudo-labels gradually decreases. This happens because lowering the bar inevitably introduces background noise into the training set. Additionally, the method relies on pre-extracted, frozen feature extractors (I3D), meaning it does not learn end-to-end representations directly from raw video pixels. Mitigating the precision drop in later stages remains an open challenge for future work.

Summary updated

Abstract

Weakly supervised temporal action localization is a challenging task as only the video-level annotation is available during the training process. To address this problem, we propose a two-stage approach to fully exploit multi-resolution information in the temporal domain and generate high quality frame-level pseudo labels based on both appearance and motion streams. Specifically, in the first stage, we generate reliable initial frame-level pseudo labels, and in the second stage, we iteratively refine the pseudo labels and use a set of selected frames with highly confident pseudo labels to train neural networks and better predict action class scores at each frame. We fully exploit temporal information at multiple scales to improve temporal action localization performance. Specifically, in order to obtain reliable initial frame-level pseudo labels, in the first stage, we propose an Initial Label Generation (ILG) module, which leverages temporal multi-resolution consistency to generate high quality class activation sequences (CASs), which consist of a number of sequences with each sequence measuring how likely each video frame belongs to one specific action class. In the second stage, we propose a Progressive Temporal Label Refinement (PTLR) framework. In our PTLR framework, two networks called Network-OTS and Network-RTS, which are respectively used to generate CASs for the original temporal scale and the reduced temporal scales, are used as two streams (i.e., the OTS stream and the RTS stream) to refine the pseudo labels in turn. By this way, the multi-resolution information in the temporal domain is exchanged at the pseudo label level, and our work can help improve each stream (i.e., the OTS/RTS stream) by exploiting the refined pseudo labels from another stream (i.e., the RTS/OTS stream).
BibTeX
@article{su2021improving,
  author = {Rui Su and Dong Xu and Luping Zhou and Wanli Ouyang},
  title = {{Improving Weakly Supervised Temporal Action Localization by Exploiting Multi-resolution Information in Temporal Domain}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
}
Figure