Similar Modality Enhancement and Action Consistency Learning for Weakly Supervised Temporal Action Localization

Maodong Li, Chao Zheng, Jian Wang, Bing LiAAAI 2025 · DOI

TL;DR

This paper introduces SEAL, a novel method for Weakly Supervised Temporal Action Localization (WTAL) that tackles the challenge of learning from video-level labels by enhancing cross-modal feature consistency and enforcing action consistency. SEAL achieves state-of-the-art performance by leveraging similar modality enhancement to create better pseudo-labels and action consistency learning to ensure complete and semantically coherent action representations.

What Problem Does This Paper Address?

Temporal Action Localization (TAL) aims to pinpoint the exact start and end times of actions within untrimmed videos. While fully supervised methods achieve high accuracy, they require meticulous frame-level annotations, which are expensive and time-consuming to obtain. Weakly Supervised Temporal Action Localization (WTAL) offers a more practical alternative by relying solely on video-level labels (e.g., "this video contains a car crash"). However, WTAL methods face two main challenges:

  1. Task Discrepancy: Existing WTAL methods often use pre-trained encoders designed for trimmed action classification. This mismatch in objectives can lead to suboptimal representations for localization tasks, especially concerning action boundaries and background frames.
  2. Action Consistency: Current methods tend to focus on distinguishing foreground from background, potentially overlooking the internal consistency of an action's temporal progression and semantic meaning. This can lead to confusion between similar actions and inaccurate localization.

Methods

SEAL addresses these challenges with two core modules:

  1. Similar Modality Enhancement (SME) Module: This module aims to mitigate task discrepancy by generating more robust pseudo-labels.
  2. Hard-global Descriptors: It constructs "hard-global" descriptors for each action category by aggregating appearance and motion features from original input data. These descriptors capture core semantic information and reduce noise.
  3. Pseudo-label Generation: Using cosine similarity, it calculates how well video features align with these descriptors. Based on the degree of consensus between RGB and flow modalities, it generates "high-confidence" and "moderate-confidence" pseudo-labels. These pseudo-labels guide the model to learn more balanced and consistent cross-modal representations.
  4. Action Consistency Learning (ACL) Module: This module focuses on improving action representation by enforcing consistency from multiple perspectives.
  5. Process Consistency (PC) Loss: This loss encourages proposals to capture the "action center" of an instance, promoting completeness of the action process and penalizing proposals that deviate too much.
  6. Semantic Consistency (SC) Loss: This loss ensures that proposals classified as the same action category within a video share similar semantic representations. It achieves this by aggregating features from consistently classified proposals into "soft-local descriptors" and aligning other proposals to these.

These modules work together within a proposal-based framework, refining segment-level features and ultimately producing more accurate temporal action localization.

Datasets

The paper evaluates SEAL on two widely used datasets for temporal action localization:

  • THUMOS14: A standard benchmark for action recognition and detection, containing 20 action categories.
  • ActivityNet1.3: A larger-scale dataset with 200 action categories, offering a more challenging evaluation.

The authors followed standard practices for training and testing on these datasets.

Key Insights and Ideas

  • Pseudo-labels from Original Features: Instead of relying on complex pseudo-labeling strategies or additional training layers, SEAL effectively generates useful pseudo-labels directly from original features. This makes the approach simpler and more lightweight.
  • Cross-Modal Consistency is Key: The paper strongly emphasizes that aligning representations from different modalities (RGB and optical flow) is crucial for overcoming task discrepancies inherent in WTAL.
  • Action Completeness Matters: Simply identifying an action isn't enough; understanding its temporal progression is vital. The PC loss directly addresses this by ensuring proposals cover the core of an action.
  • Semantic Coherence Enhances Discrimination: For actions that are visually similar, enforcing semantic consistency among their proposals helps the model differentiate them more effectively, leading to better localization.
  • Generalization to Different Proposals: SEAL demonstrates robust performance even when using candidate proposals generated by other methods, highlighting its generalizability.

Key Results

SEAL achieves state-of-the-art performance on both THUMOS14 and ActivityNet1.3 datasets.

  • On THUMOS14, SEAL significantly outperforms existing WTAL methods, achieving a notable improvement in average [email protected]:0.7. For instance, it shows a 3.2% improvement over the previous best method.
  • On ActivityNet1.3, SEAL also demonstrates strong competitiveness, outperforming comparison methods on average [email protected]:0.95.
  • Ablation studies confirm the effectiveness of each proposed module (SME and ACL) and their individual loss functions (PC and SC), showing that they all contribute to performance gains without conflicts. The video-level aggregation for descriptors also proves more effective than segment-level.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while their approach is simpler than some, it still relies on pre-trained feature extractors.
  • The performance of the moderate-confidence pseudo-labels is noted to be potentially affected by unforeseen uncertainties, and their reliance is reduced over training epochs.
  • Future Work:
    • The authors suggest that their method's ability to generalize to candidate proposals from other methods opens avenues for further exploration in combining SEAL with various proposal generation techniques.
  • The focus on cross-modal consistency and action consistency learning could be extended to other video understanding tasks.

Summary updated

BibTeX
@inproceedings{li2025similar,
  author = {Maodong Li and Chao Zheng and Jian Wang and Bing Li},
  title = {{Similar Modality Enhancement and Action Consistency Learning for Weakly Supervised Temporal Action Localization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32512},
}
Figure