Estimation of Reliable Proposal Quality for Temporal Action Detection

Liansheng Zhuang

TL;DR

This paper tackles the problem of temporal action detection (TAD) by proposing a novel method, BREM, that aligns the classification and localization tasks in anchor-free models. BREM achieves state-of-the-art results on THUMOS14 by effectively estimating proposal quality from both "moment" and "region" perspectives, leading to more accurate action localization.

What Problem Does This Paper Address?

Temporal Action Detection (TAD) is the task of identifying and localizing specific actions within untrimmed videos. Recent advances in anchor-free methods have shown great promise, but they often struggle with a fundamental issue: the temporal misalignment between the classification (what action is it?) and localization (where exactly does it start and end?) branches. This misalignment means that an action might be correctly classified but poorly localized, or vice-versa. The paper highlights that the classification score, which is typically used to rank proposals, doesn't always reflect the true localization quality (measured by temporal Intersection over Union or tIoU). This can lead to good proposals being discarded due to inaccurate scores, hindering overall performance.

Methods

The core contribution is the Boundary Evaluate Module (BEM) and the Region Evaluate Module (REM), collectively called BREM.

  • Boundary Evaluate Module (BEM): This module focuses on the "moment" perspective. It estimates the quality of action boundaries by analyzing local appearance and motion changes over time. A key innovation here is the use of multi-scale boundary quality. Recognizing that actions have varying durations, BEM employs multiple anchor sizes to assess boundary quality, making it more robust to actions of different lengths. This is a departure from single-scale approaches that can be overly sensitive to boundary errors for short actions or miss them for long ones.
  • Region Evaluate Module (REM): This module takes a "region" perspective. It refines action proposals by using a novel sampling method to extract richer contextual features from within the proposal. Instead of just using point features, REM samples features at three key locations within a proposal and aggregates them. This enhanced feature representation helps in more accurately estimating the category score and refining the proposal's boundary.

BREM is designed to be generic and plug-and-play, meaning it can be easily integrated with existing anchor-free TAD frameworks to improve their performance.

Datasets

The paper evaluates its method on two widely used datasets:

  • THUMOS14: A benchmark dataset for temporal action detection.
  • ActivityNet-1.3: Another large-scale dataset for human activity understanding.

The authors train their models on the validation set and report results on the testing set for THUMOS14, and on the validation set for ActivityNet-1.3.

Key Insights and Ideas

  • Misalignment is a Key Bottleneck: The paper convincingly demonstrates that the temporal misalignment between classification and localization is a significant hurdle for current anchor-free TAD methods.
  • Proposal Quality is Crucial: The classification score is a poor proxy for actual localization quality (tIoU). Improving the estimation of proposal quality is essential for better performance.
  • Multi-scale Boundary Quality is Necessary: Actions of different durations require different sensitivities to boundary errors. A multi-scale approach to boundary quality assessment is more effective than single-scale methods.
  • Context Matters for Localization: Incorporating richer contextual features from within a proposal (region perspective) significantly aids in refining localization accuracy.
  • Modularity is Powerful: The BREM modules are designed to be easily integrated, showing that combining these complementary "moment" and "region" views can boost performance across different base architectures.

Key Results

  • State-of-the-Art on THUMOS14: When integrated with the ActionFormer framework, BREM achieved a new state-of-the-art performance of 63.6% average mAP on THUMOS14.
  • Significant Improvements: BREM improved the performance of a baseline anchor-free framework on THUMOS14 by 3.6% (from 53.1% to 56.7% average mAP).
  • Competitive Results on ActivityNet-1.3: A competitive result of 36.2% average mAP was achieved on ActivityNet-1.3 when combined with ActionFormer.
  • Effectiveness of Components: Ablation studies showed that both BEM and REM contribute significantly, with their combination yielding the best results. BEM improved average mAP by 2.2%, and REM by 1.2%.
  • Efficiency: BREM adds negligible computational overhead, with a slight increase in inference time and memory usage, demonstrating its practical viability.

Limitations and Future Work

  • Hyperparameter Sensitivity: While multi-scale boundary quality is less sensitive than single-scale, the choice of anchor scales and mapping coefficients (τ) still influences performance, as explored in the ablation studies.
  • Future Work: The paper suggests that BREM's generic nature makes it suitable for integration with other TAD methods, implying potential for further performance gains across a wider range of architectures. The authors also highlight the importance of further exploring feature extraction methods for REM.

Summary updated

Abstract

Temporal action detection (TAD) aims to locate and recognize the actions in an untrimmed video. Anchor-free methods have made remarkable progress which mainly formulate TAD into two tasks: classification and localization using two separate branches. This paper reveals the temporal misalignment between the two tasks hindering further progress. To address this, we propose a new method that gives insights into moment and region perspectives simultaneously to align the two tasks by acquiring reliable proposal quality. For the moment perspective, Boundary Evaluate Module (BEM) is designed which focuses on local appearance and motion evolvement to estimate boundary quality and adopts a multi-scale manner to deal with varied action durations. For the region perspective, we introduce Region Evaluate Module (REM) which uses a new and efficient sampling method for proposal feature representation containing more contextual information compared with point feature to refine category score and proposal boundary. The proposed B oundary Evaluate Module and R egion E valuate M odule (BREM) are generic, and they can be easily integrated with other anchor-free TAD methods to achieve superior performance. In our experiments, BREM is combined with two different frameworks and improves the performance on THUMOS14 by 3.6% and 1.0% respectively, reaching a new state-of-the-art (63.6% average mAP). Meanwhile, a competitive result of 36.2% average mAP is achieved on ActivityNet-1.3 with the consistent improvement of BREM. The codes are released at \urlhttps://github.com/Junshan233/BREM.
BibTeX
@inproceedings{zhuang2022estimation,
  author = {Liansheng Zhuang},
  title = {{Estimation of Reliable Proposal Quality for Temporal Action Detection}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=4EgwrTG3FW},
}
Figure