Abstract
Temporal action detection, a critical branch of video analysis, aims to locate the precise start and end points of actions and classify them into correct categories. A key challenge in this task is generating high-quality temporal proposals. In this paper, we introduce the Multi-Scale Proposal Regression Network (MPRN). Our approach first encodes visual features to predict action scores, grouping them into initial rough proposals. We then refine these boundaries using a novel multi-scale regression strategy. Unlike traditional methods, MPRN utilizes flexible boundary segments that adapt to varying action lengths. Experimental results on the THUMOS-14 and ActivityNet-v1.3 datasets demonstrate that MPRN outperforms existing state-of-the-art methods, exhibits strong generalization to unseen action classes, and maintains high performance regardless of the specific visual feature encoder used.
Introduction
In video analysis, Temporal Action Detection requires determining exactly when an action occurs. This process typically involves two steps: proposal generation (finding candidate segments) and classification (identifying the action). The primary bottleneck is often the proposal stage. Human actions vary significantly in duration, making fixed-scale "sliding window" methods inefficient or inaccurate. Furthermore, traditional "watershed" algorithms may miss valid actions if confidence scores dip momentarily. MPRN addresses these limitations by creating a flexible, scale-agnostic framework that recovers missed proposals through peak-point detection and refines them via multi-scale regression.
The MPRN Framework
The MPRN architecture is divided into three distinct stages designed to maximize both sensitivity and precision in action detection.
1. Visual Feature Encoding
The system employs a "two-stream" network to process video snippets. A spatial stream analyzes individual frames to capture appearance, while a temporal stream utilizes optical flow to capture motion. These streams are integrated into comprehensive feature vectors.
2. Action Score Computation
A two-layer Convolutional Neural Network (CNN) predicts the probability of an action occurring at any given time point. Beyond simple thresholding, the network identifies "peak points" where scores change sharply. This "supplementary proposal" strategy ensures that actions with clear boundaries are captured even if their overall confidence scores are relatively low.
3. Multi-Scale Proposal Regression
This is the core innovation of the MPRN. Once rough proposals are identified, the network examines the start and end segments at three different scales. Fully connected layers then regress (fine-tune) these boundaries. This multi-scale approach allows the network to "zoom" in or out, providing the flexibility needed to handle the diverse durations of human activities.
Experimental Evaluation
MPRN was evaluated on two major benchmarks, showing significant improvements over previous baselines:
- ActivityNet-v1.3: MPRN achieved an Average Recall (AR@100) of 74.38%, a substantial increase over methods like TAG (63.52%) and TURN (63.46%).
- THUMOS-14: The model nearly doubled the performance of the TAG baseline at low proposal counts (AR@50), reaching 37.11%.
- Generalization: The model demonstrated a remarkable ability to detect actions it had never seen before. On unseen classes in THUMOS-14, it maintained an AR@100 of 42.34%, suggesting it learns a general "grammar" of action boundaries rather than just memorizing specific movements.
Conclusion
The Multi-Scale Proposal Regression Network (MPRN) significantly advances the state-of-the-art in temporal action proposal generation. By combining peak-point sensitivity with multi-scale boundary refinement, it offers a robust and flexible solution for complex video analysis. While MPRN is highly effective at finding actions, future work will focus on further increasing boundary precision to match specialized matching models at very high overlap thresholds.
Summary updated