Two-Stream Convolutional Networks for Action Recognition in Videos

Karen Simonyan, Andrew Zisserman

TL;DR

Simonyan and Zisserman bridge the gap between deep learning and hand-crafted features in video analysis by introducing a two-stream architecture. By decoupling a video into a spatial stream (appearance) and a temporal stream (motion via stacked optical flow), they achieved state-of-the-art results that significantly outperformed previous "3D" or "stacked frame" deep learning approaches.

The Challenge of Video Classification

In 2014, Deep Convolutional Neural Networks (ConvNets) had already revolutionized static image recognition, largely thanks to architectures like AlexNet and massive datasets like ImageNet. However, applying these same networks to video proved surprisingly difficult.

The core challenge is that video is not merely a sequence of static images; it contains a vital temporal dimension. Previous attempts at "deep" action recognition usually involved stacking raw video frames as input, hoping the network would implicitly learn to track motion. These models performed poorly, often failing to beat "shallow" hand-crafted features like Improved Dense Trajectories (IDT). The authors set out to find a ConvNet architecture that could explicitly capture both the "what" (objects and scenes) and the "how" (motion) of an action.

The Two-Stream Architecture

The authors propose a Two-Stream ConvNet architecture, inspired by the biological hypothesis that the human visual cortex has two distinct pathways: the ventral stream for object recognition and the dorsal stream for motion recognition.

Two-Stream ConvNet Architecture

This architecture splits the video processing into two independent networks:

  1. Spatial Stream ConvNet: This is a standard image classification network that operates on individual RGB video frames. It captures the static appearance of the scene and the objects involved (e.g., a "bow" or a "basketball court"). Because it is essentially a standard image classification task, the authors could pre-train this stream on the massive ImageNet dataset to prevent overfitting.
  2. Temporal Stream ConvNet: Instead of raw pixels, this stream takes stacked dense optical flow as input. Optical flow represents the displacement of pixels between consecutive frames. By stacking flow fields from $L$ consecutive frames (typically $L=10$), the network receives an explicit representation of motion over time.
Optical Flow Visualization
  1. Late Fusion: The two streams are trained independently. At test time, their softmax scores are combined—either by simple averaging or by training a linear SVM—to produce the final action classification.
  2. Multi-task Learning: To combat the relative scarcity of video data compared to images, the authors used a multi-task learning framework to train the temporal network on two different datasets simultaneously, effectively increasing the training signal.

Key Insights and Ideas

  • Explicit Motion is Key: The "aha moment" of this paper is the realization that ConvNets struggle to learn motion from raw pixels if the dataset is small. By providing optical flow as a pre-computed input, the authors "hand-fed" the motion information to the network, allowing it to focus on high-level motion patterns rather than basic pixel-tracking.
  • Decoupling Works: Separating appearance from motion allows for different optimization strategies. For example, the spatial stream benefits immensely from ImageNet pre-training, while the temporal stream requires aggressive dropout (up to 0.9) to avoid overfitting on the smaller video datasets.
  • Motion is More Discriminative: Interestingly, the temporal stream alone (83.7% accuracy on UCF-101) significantly outperformed the spatial stream alone (73.0%), confirming that motion is often a stronger cue for action than appearance.

Key Results

The models were evaluated on the two primary benchmarks for action recognition at the time: UCF-101 (13,320 videos, 101 categories) and HMDB-51 (6,800 videos, 51 categories).

  • State-of-the-Art Performance: The two-stream model achieved 88.0% accuracy on UCF-101 and 59.4% on HMDB-51.
  • Beating Previous Deep Nets: It outperformed the previous best deep learning attempt (the "Slow Fusion" ConvNet) by a massive margin—roughly a 23% absolute improvement on UCF-101.
  • Competitive with Hand-crafted Features: For the first time, a deep learning model was able to match or exceed the performance of the highly optimized, hand-crafted Improved Dense Trajectory (IDT) pipelines.

Limitations and Future Work

While groundbreaking, the two-stream approach has limitations:

  • Short-term Focus: The temporal stream only looks at a short window (about 0.4 seconds of video). It cannot capture long-term dependencies or the "story" of a complex, multi-stage action.
  • Preprocessing Bottleneck: Computing dense optical flow is computationally expensive and must be done before the network can run, making real-time deployment difficult.
  • Camera Motion: While the authors used "mean flow subtraction" to mitigate camera movement, the temporal stream can still be confused by significant global motion that isn't related to the human action.

The authors suggest that incorporating explicit camera motion compensation and exploring ways to pool features over longer trajectories could further improve the model, paving the way for the next generation of video understanding networks.

Summary updated

Abstract

We investigate architectures of discriminatively trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. We also aim to generalise the best performing hand-crafted features within a data-driven learning framework. Our contribution is three-fold. First, we propose a two-stream ConvNet architecture which incorporates spatial and temporal networks. Second, we demonstrate that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, we show that multi-task learning, applied to two different action classification datasets, can be used to increase the amount of training data and improve the performance on both. Our architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.
BibTeX
@inproceedings{simonyan2014two,
  author = {Karen Simonyan and Andrew Zisserman},
  title = {{Two-Stream Convolutional Networks for Action Recognition in Videos}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=ryWhCvbd-H},
}
Figure