Abstract
Temporal action proposal generation is a critical task in video understanding, traditionally dominated by sliding window or grouping-based paradigms. This paper introduces Deep Point-wise Prediction (DPP), an anchor-free framework that predicts action existence and temporal offsets directly for every point in a temporal feature map. By leveraging a Temporal Feature Pyramid Network (TFPN), DPP handles actions of varying lengths efficiently. Evaluated on the THUMOS14 dataset, DPP achieves state-of-the-art accuracy, outperforming previous methods by significant margins while maintaining a high processing speed of over 1,000 frames per second.
Introduction
Temporal action proposal generation involves identifying the start and end times of actions in untrimmed videos. Historically, researchers have relied on two main approaches:
- Sliding Windows: Evaluating fixed-length intervals across the video, which is computationally expensive and struggles with actions that do not fit predefined lengths.
- Actionness Grouping: Scoring every frame for "actionness" and grouping high-scoring frames via complex heuristics, which often limits precision.
DPP moves away from these paradigms, mimicking human perception by estimating action boundaries from single temporal glimpses rather than exhaustive window searches.
Methods
The DPP framework consists of three core components designed for efficiency and scale-invariance:
- Backbone Network: Extracts high-level spatio-temporal features using 3D convolutional networks (e.g., C3D or 3D-ResNet).
- Temporal Feature Pyramid Network (TFPN): Inspired by image-based Feature Pyramid Networks, this component constructs a hierarchy of feature maps. Each level has a different receptive field, allowing the model to detect short actions at lower levels and long actions at higher levels.
- Prediction Network: For every point in these feature maps, the model performs point-wise classification (foreground vs. background) and regression (relative offsets to the start and end times).
Crucially, DPP is anchor-free. It treats every point in the temporal feature map as a potential action center and predicts boundaries relative to that point. To simplify optimization, the authors introduced a "Scale Assignment" strategy, ensuring each pyramid level only predicts actions that fall within a specific range of its receptive field.
Key Results
The model was evaluated on the THUMOS 2014 dataset, a standard benchmark for temporal localization.
- Blazing Speed: When using a ResNet-101 backbone, DPP runs at 1,294 FPS on a single GTX 1080 Ti, significantly outpacing real-time requirements.
- State-of-the-Art Accuracy: DPP outperformed previous sliding-window methods like TURN by a massive 16.2% in Average Recall (AR@200). It also surpassed BSN, a popular grouping-based method, by 2% in AR@200.
Conclusion and Future Work
DPP demonstrates that point-wise prediction is a superior, more efficient alternative to traditional proposal generation methods. While the performance is currently tied to the quality of the backbone features, the architecture's stability and speed make it a strong candidate for real-world applications. Future work may involve integrating DPP into an end-to-end detection pipeline that predicts specific action categories alongside temporal intervals.
The code for this project is available at: https://github.com/liluxuan1997/DPP
Summary updated