Abstract
In this paper, we present YoTube—a novel network fusion framework for searching action proposals in untrimmed videos, where each action proposal corresponds to a spatialtemporal video tube that potentially locates one human action. Our method consists of a recurrent YoTube detector and a static YoTube detector, where the recurrent YoTube explores the regression capability of RNN for candidate bounding boxes predictions using learnt temporal dynamics and the static YoTube produces the bounding boxes using rich appearance cues in a single frame. Both networks are trained using rgb and optical flow in order to fully exploit the rich appearance, motion and temporal context, and their outputs are fused to produce accurate and robust proposal boxes. Action proposals are finally constructed by linking these boxes using dynamic programming with a novel trimming method to handle the untrimmed video effectively and efficiently. Extensive experiments on the challenging UCF-101 and UCF-Sports datasets show that our proposed technique obtains superior performance compared with the state-of-the-art.
Introduction
While action classification—identifying what is happening in a video—has seen massive leaps, action detection—identifying exactly where and when it happens—remains a significant challenge. Most existing methods struggle with "untrimmed" videos, which are long clips where the action only occupies a small fraction of the total time and space.
Current research often suffers from two main gaps:
- Isolated Learning: Spatial appearance and temporal motion are frequently separated, leading to accumulated errors and preventing end-to-end optimization.
- Noise in Untrimmed Data: Standard proposal methods often generate "tubes" that span the entire video length, including significant background noise and non-action frames.
YoTube addresses these by creating a unified framework that learns appearance, motion, and temporal context simultaneously to produce spatially compact and temporally smooth action proposals.
Methodology
The YoTube framework consists of three main stages: Dual-Detector Fusion, Action Path Linking, and Peak Trimming.
Dual-Detector Fusion
The core of the system is the combination of two specialized detectors:
- Recurrent YoTube: This adapts the Long-term Recurrent Convolutional Network (LRCN) architecture. Unlike standard RNNs used for classification, this model explores the regression capability of LSTMs to predict bounding box coordinates $(x, y, w, h)$ across frames, effectively tracking the action using temporal context.
- Static YoTube: A CNN-based detector (inspired by YOLO) that focuses on global appearance cues within individual frames. This helps reduce interference from background clutter that might confuse a purely temporal model.
Both detectors process two streams of data: RGB (for appearance) and Optical Flow (for motion). Their outputs are fused to generate robust candidate boxes for every frame.
Action Path Linking
To turn individual boxes into a continuous "tube," the authors use a dynamic programming approach (the Viterbi algorithm). They define a score for each path based on the "actionness" confidence of the boxes and the spatial overlap (IoU) between successive frames, ensuring the resulting path is both high-quality and physically smooth.
Peak Trimming
Linked paths often span the whole video and must be trimmed to the actual start and end of the action. The authors observed that transitions in "background scores" typically signify the start or end of an action. By detecting local peaks in actionness and background scores, the algorithm identifies the most likely sub-sequences, effectively cutting out the "noise" of untrimmed footage.
Experiments and Results
The authors evaluated YoTube on two primary benchmarks: UCF-101 (Localization Subset) and UCF-Sports. UCF-101 is particularly challenging as about 25% of the videos are untrimmed.
YoTube significantly outperformed existing state-of-the-art (SOTA) methods:
- UCF-101: Achieved a 59.19% recall at a 0.5 IoU threshold, compared to 39.64% for the previous SOTA.
- UCF-Sports: Achieved a 97.87% recall, surpassing previous benchmarks.
- Trimming Impact: On untrimmed UCF-101 videos, the peak trimming method improved recall by nearly 7% compared to using untrimmed paths, proving its effectiveness in filtering out temporal noise.
Conclusion
YoTube demonstrates the power of fusing recurrent and static regression networks for action localization. By leveraging LSTMs for coordinate regression and introducing an efficient peak trimming algorithm, the framework achieves superior performance on untrimmed video data. Future work includes exploring more complex temporal modeling and extending the framework to handle multi-person action proposals in crowded environments.
Summary updated