Abstract
Two-stream convolutional networks have shown strong performance in video action recognition for their ability to capture spatial and temporal features simultaneously. However, the calculation of optical flow is time-consuming and cannot be applied to real-time processing. To address this, we propose SpatioTemporal Relation Networks (STRN), an end-to-end architecture that extracts spatiotemporal information using only RGB input. STRN consists of an appearance stream and a motion stream. The motion stream takes relation information between adjacent features in the appearance stream as input via "relation blocks." Validated on UCF-101 and HMDB-51, STRN achieves superior performance without the computational tax of optical flow.
Introduction
Recognizing actions in video is a fundamental challenge in computer vision. While identifying objects in static images is well-understood, capturing the temporal dynamics of actions like "brushing teeth" or "playing guitar" typically requires modeling motion. The traditional solution, the two-stream network, relies on optical flow to represent motion. However, optical flow calculation is a massive computational bottleneck, often limiting systems to low frame rates unsuitable for real-time use.
SpatioTemporal Relation Networks (STRN) offer a "middle way." By extracting motion cues directly from spatial features using novel relation blocks, STRN eliminates the need for optical flow while maintaining high accuracy and achieving a processing speed of 410 frames per second (fps).
Methodology: The STRN Architecture
The core of STRN is a dual-stream architecture where both branches derive information from raw RGB frames.
Dual-Stream Design
- Appearance Stream: This branch retains the structure of a standard spatial stream (e.g., ResNet-50) but processes consecutive RGB frames to capture evolving spatial context.
- Motion Stream: Instead of processing optical flow, this stream takes "relation information" as input. This information is extracted from the appearance stream's intermediate features.
Relation Blocks
The "Relation Block" is the primary innovation of this work. It acts as a bridge, calculating the relationship between features of adjacent frames. The authors found that high-level features extracted by 2D CNNs already contain implicit motion data.
While experiments were conducted with Average Pooling and Max Pooling, the Conv3D relation block proved most effective. It utilizes a small 3D convolution to learn the spatiotemporal relationships between features, which are then integrated into the motion stream via residual connections.
Experimental Results
The model was evaluated on the UCF-101 and HMDB-51 datasets.
Accuracy and Efficiency
On the UCF-101 benchmark, STRN (using RGB and RGB differences) achieved an accuracy of 93.2%. This outperforms traditional 3D CNN models like C3D (85.8%) and P3D (88.6%).
The most significant advantage is speed. While traditional Temporal Segment Networks (TSN) using optical flow operate at roughly 14 fps, STRN runs at 410 fps. This efficiency stems from leveraging pre-trained 2D weights and avoiding the parameter explosion of deep 3D kernels.
Key Insights
- Early Integration: Relation blocks are most effective when placed in the early layers of the network. Fine-grained motion information is more accessible in low-level features before they become too semantically abstract.
- RGB Differences: Simply including the difference between consecutive frames as a supplementary input provides a significant performance boost at almost zero computational cost.
Conclusion
SpatioTemporal Relation Networks demonstrate that explicit optical flow is not a prerequisite for high-performance action recognition. By intelligently extracting relationships between spatial features, STRN provides a real-time, end-to-end solution for video analysis. Future work will explore more complex relation extractors and scaling the architecture to massive datasets like Kinetics.
Summary updated