Spatiotemporal Relation Networks for Video Action Recognition

Zheng Liu, Haifeng HuIEEE Access 2019 · OpenReview

Abstract

Two-stream convolutional networks have shown strong performance in video action recognition for their ability to capture spatial and temporal features simultaneously. However, the calculation of optical flow is time-consuming and cannot be applied to real-time processing. To address this, we propose SpatioTemporal Relation Networks (STRN), an end-to-end architecture that extracts spatiotemporal information using only RGB input. STRN consists of an appearance stream and a motion stream. The motion stream takes relation information between adjacent features in the appearance stream as input via "relation blocks." Validated on UCF-101 and HMDB-51, STRN achieves superior performance without the computational tax of optical flow.

Introduction

Recognizing actions in video is a fundamental challenge in computer vision. While identifying objects in static images is well-understood, capturing the temporal dynamics of actions like "brushing teeth" or "playing guitar" typically requires modeling motion. The traditional solution, the two-stream network, relies on optical flow to represent motion. However, optical flow calculation is a massive computational bottleneck, often limiting systems to low frame rates unsuitable for real-time use.

SpatioTemporal Relation Networks (STRN) offer a "middle way." By extracting motion cues directly from spatial features using novel relation blocks, STRN eliminates the need for optical flow while maintaining high accuracy and achieving a processing speed of 410 frames per second (fps).

Methodology: The STRN Architecture

The core of STRN is a dual-stream architecture where both branches derive information from raw RGB frames.

The STRN architecture featuring Appearance and Motion streams connected by Relation Blocks.

Dual-Stream Design

  1. Appearance Stream: This branch retains the structure of a standard spatial stream (e.g., ResNet-50) but processes consecutive RGB frames to capture evolving spatial context.
  2. Motion Stream: Instead of processing optical flow, this stream takes "relation information" as input. This information is extracted from the appearance stream's intermediate features.

Relation Blocks

The "Relation Block" is the primary innovation of this work. It acts as a bridge, calculating the relationship between features of adjacent frames. The authors found that high-level features extracted by 2D CNNs already contain implicit motion data.

Detailed view of the Conv3D relation block used to bridge the appearance and motion streams.

While experiments were conducted with Average Pooling and Max Pooling, the Conv3D relation block proved most effective. It utilizes a small 3D convolution to learn the spatiotemporal relationships between features, which are then integrated into the motion stream via residual connections.

Experimental Results

The model was evaluated on the UCF-101 and HMDB-51 datasets.

Accuracy and Efficiency

On the UCF-101 benchmark, STRN (using RGB and RGB differences) achieved an accuracy of 93.2%. This outperforms traditional 3D CNN models like C3D (85.8%) and P3D (88.6%).

The most significant advantage is speed. While traditional Temporal Segment Networks (TSN) using optical flow operate at roughly 14 fps, STRN runs at 410 fps. This efficiency stems from leveraging pre-trained 2D weights and avoiding the parameter explosion of deep 3D kernels.

Key Insights

  • Early Integration: Relation blocks are most effective when placed in the early layers of the network. Fine-grained motion information is more accessible in low-level features before they become too semantically abstract.
  • RGB Differences: Simply including the difference between consecutive frames as a supplementary input provides a significant performance boost at almost zero computational cost.

Conclusion

SpatioTemporal Relation Networks demonstrate that explicit optical flow is not a prerequisite for high-performance action recognition. By intelligently extracting relationships between spatial features, STRN provides a real-time, end-to-end solution for video analysis. Future work will explore more complex relation extractors and scaling the architecture to massive datasets like Kinetics.

Summary updated

Abstract

Two-stream convolutional networks have shown strong performance in a video action recognition task for its ability to capture spatial and temporal features simultaneously. However, the calculation of optical flow is time-consuming and it cannot be applied to the real-time processing of video. To address this problem, this paper proposes a new end-to-end architecture called SpatioTemporal Relation Networks (STRN) to extract spatial information and temporal information simultaneously from the video with the only RGB input. STRN consist of two branches, called appearance stream and motion stream, respectively. Appearance stream retains the structure of the original spatial stream in the two-stream architecture with the input of consecutive frames instead of a single frame. Motion stream, which takes relation information between the adjacent features in the appearance stream as an input, can effectively complement appearance stream. A relation block is an extractor which is used to extract relation information from the appearance stream. STRN can learn spatiotemporal information from the video with the only RGB input, which avoids the calculation of optical flow. We validate the STRN on UCF-101 and HMDB-51 and achieve better performance.
BibTeX
@inproceedings{liu2019spatiotemporal,
  author = {Zheng Liu and Haifeng Hu},
  title = {{Spatiotemporal Relation Networks for Video Action Recognition}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=JHFDmfUlMy},
}
Figure