Abstract
Temporal Action Detection (TAD) involves identifying the start and end times of actions in untrimmed videos alongside their categorical labels. While anchor-free methods have gained popularity for their simplicity, they often suffer from "temporal misalignment," where classification confidence scores do not accurately reflect localization quality. This paper introduces the Boundary and Region Evaluate Module (BREM), a plug-and-play framework designed to align these distributions. By integrating a multi-scale boundary evaluator with a regional feature refiner, BREM achieves a new state-of-the-art result of 63.6% average mAP on the THUMOS14 benchmark.
Introduction
Modern Temporal Action Detection (TAD) frameworks typically decompose the task into two parallel sub-problems: classification (identifying the action) and localization (predicting temporal boundaries). However, a critical bottleneck in anchor-free detectors is the lack of correlation between these two outputs. A model may produce a high classification score for a poorly localized segment, or vice versa. This misalignment causes high-quality proposals to be suppressed by lower-quality ones during post-processing steps like Non-Maximum Suppression (NMS).
An "Oracle" experiment highlights the severity of this issue: if classification scores were perfectly aligned with temporal Intersection over Union (tIoU), performance on standard benchmarks could nearly double, jumping from 43.7% to 85.8% mAP. This suggests that improving quality estimation is more impactful than simply refining boundary predictions.
Proposed Method: BREM
The Boundary and Region Evaluate Module (BREM) is a generic framework that can be integrated into existing anchor-free detectors. It addresses misalignment from two perspectives: the temporal moment and the regional context.
Boundary Evaluate Module (BEM)
The BEM focuses on the "moment" perspective by predicting the quality of start and end boundaries. Recognizing that temporal precision requirements vary with action duration—where a one-second error is critical for short actions but negligible for long ones—BEM employs a multi-scale approach. It evaluates boundaries across multiple temporal scales, ensuring the model remains sensitive to both short, dense actions and long, sparse ones.
Region Evaluate Module (REM)
While standard detectors often rely on single-point features, the REM adopts a "region" perspective. It samples features from three key locations within a proposal: the start, center, and end. These features are aggregated to provide a holistic view of the action, which is then used to refine the final category score and boundary offsets. This regional context ensures that the final quality estimation is based on the entire action instance rather than a single frame.
Experiments and Results
The effectiveness of BREM was evaluated on two major benchmarks: THUMOS14, characterized by short, dense actions, and ActivityNet-1.3, which contains longer, more varied action durations.
- State-of-the-Art Performance: When combined with the ActionFormer architecture, BREM reached 63.6% average mAP on THUMOS14, setting a new benchmark for the field.
- Genericity: Adding BREM to a standard anchor-free baseline resulted in a 3.6% absolute mAP improvement, demonstrating its utility as a plug-and-play module.
- Efficiency: The module is computationally efficient. In testing, BREM only added approximately 10ms to the inference time (increasing from 45.2ms to 55.6ms), making it suitable for practical deployment.
Conclusion
BREM effectively mitigates the temporal misalignment problem in anchor-free TAD by combining multi-scale boundary evaluation with regional feature refinement. While the module significantly boosts performance, its efficacy remains partially constrained by the underlying feature backbone. Future research may focus on tighter integration between quality estimation and the initial feature extraction process to further enhance localization accuracy in complex video environments.
Summary updated