Abstract
This paper studies the joint learning of action recognition and temporal localization in long, untrimmed videos. We employ a multi-task learning framework that performs the three highly related steps of action proposal, action recognition, and action localization refinement in parallel instead of the standard sequential pipeline that performs the steps in order. We develop a novel temporal actionness regression module that estimates what proportion of a clip contains action. We use it for temporal localization but it could have other applications like video retrieval, surveillance, summarization, etc. We also introduce random shear augmentation during training to simulate viewpoint change. We evaluate our framework on three popular video benchmarks. Results demonstrate that our joint model is efficient in terms of storage and computation in that we do not need to compute and cache dense trajectory features, and that it is several times faster than its sequential ConvNets counterpart. Yet, despite being more efficient, it outperforms state-of-the-art methods with respect to accuracy.
Introduction
Most video analysis research focuses on "trimmed" clips—short snippets where the action is already centered. However, real-world video is typically "untrimmed," containing long sequences of background or non-action footage. Detecting specific actions within these messy streams requires both identifying the action type and precisely locating its temporal boundaries.
The conventional approach to this problem is a sequential pipeline: first, generate candidate proposals; second, classify those candidates; and third, refine the start and end times. This approach suffers from two major drawbacks. First, it creates an "upper-bound" problem: if the proposal stage fails to identify a segment, the classifier never has a chance to correct it. Second, training these stages independently prevents the model from sharing features across related tasks, leading to computational inefficiency and high storage requirements for intermediate features.
Methodology: A Parallel Multi-Task Framework
To address these limitations, we propose a joint learning framework based on a 3D Convolutional Neural Network (C3D). Instead of a linear chain, our architecture uses a shared backbone that feeds into three parallel branches, as shown in Figure 1.
Parallel Branches
- Proposal Branch: A binary classifier that distinguishes between "actions of interest" and background footage.
- Classification Branch: A multi-class module identifying specific action categories (e.g., "Baseball Pitch"). To improve feature learning and prevent overfitting, we incorporate intermediate losses at the
conv5,fc6, andfc8layers. This forces the network to capture both low-level motion patterns and high-level semantic information. - Regression Branch: This branch introduces the concept of Temporal Actionness. Rather than predicting a simple overlap metric like temporal Intersection over Union (tIoU), this module predicts the proportion of a clip that contains an action. This fractional value (0 to 1) is more stable and easier for the network to learn than the highly variable tIoU.
Data Augmentation
To enhance the model's robustness to varying camera angles, we introduced Random Shear Augmentation during training. By slightly distorting frames to simulate viewpoint changes, the model learns more generalized features that are less dependent on specific camera placements.
Evaluation and Results
We evaluated our framework on three benchmarks: THUMOS 2014 (complex sports videos), MSR Actions II (basic detection), and MPII Cooking (fine-grained actions like "peeling").
Performance and Efficiency
Our model achieved a Mean Average Precision (mAP) of 19.0% on THUMOS 2014 at an overlap threshold of 0.5, surpassing previous state-of-the-art methods (see Figure 5).
Beyond accuracy, the parallel architecture offers significant practical advantages:
- Speed: The framework is several times faster than sequential ConvNet models, with training iterations taking approximately one second.
- Storage: By using a single backbone and avoiding the need to cache massive "dense trajectory" features, the final model size is less than 0.35GB.
- Robustness: The multi-task approach acts as a form of regularization. Even when the amount of trimmed training data was reduced by 50%, our model maintained high performance, whereas sequential models typically see a sharp decline in accuracy.
Conclusion
By performing proposal, classification, and localization refinement in parallel, our multi-task framework overcomes the bottlenecks of traditional sequential pipelines. The introduction of temporal actionness regression provides a more effective signal for localization than standard overlap metrics. Future work will explore end-to-end optimization similar to Faster R-CNN for video and the integration of deeper backbones like 3D Residual Networks to further enhance feature extraction.
Summary updated