Two-Stream Convolutional Networks for Action Recognition in Videos

Karen Simonyan, Andrew Zisserman

TL;DR

This paper introduces a "two-stream" convolutional neural network (CNN) architecture that significantly improves video action recognition by jointly processing appearance (spatial stream) and motion (temporal stream). By decoupling these streams and leveraging pre-training on image datasets, they achieve state-of-the-art results on challenging video action benchmarks, outperforming previous deep learning approaches.

What Problem Does This Paper Address?

Recognizing human actions in videos is a complex task. While static images can provide clues about actions (e.g., a person holding a tennis racket might be playing tennis), motion is often crucial for disambiguation and accurate recognition. Traditional approaches often struggled to effectively integrate appearance and motion information. Previous attempts to use deep CNNs for video recognition, by feeding them stacks of raw frames, yielded results worse than hand-crafted features, suggesting that deep networks might not implicitly learn motion well from raw pixels. This paper aims to bridge this gap by proposing a novel architecture that explicitly leverages both appearance and motion information in a complementary manner.

Methods

The core innovation is the two-stream architecture. This involves:

  • Spatial Stream: A standard CNN designed for image recognition, trained on individual video frames. This stream captures appearance information. Crucially, it's pre-trained on the large ImageNet dataset, a common practice for image classification that significantly boosts performance and reduces the need for massive video-specific training data.
  • Temporal Stream: A separate CNN designed to process motion. Instead of raw frames, this stream takes dense optical flow as input. Optical flow represents the apparent motion of pixels between consecutive frames. The authors explore different ways to represent this flow, including stacking multiple frames' flow fields (optical flow stacking) and sampling flow along detected trajectories (trajectory stacking). They also investigate using bi-directional flow and subtracting the mean flow to mitigate camera motion effects.
  • Late Fusion: The outputs (softmax scores) from the spatial and temporal streams are combined using either simple averaging or a linear SVM classifier. This late fusion allows each stream to learn independently before their predictions are merged.
  • Multi-task Learning: To address the issue of relatively small video datasets, the authors propose multi-task learning. This involves training a single network to perform classification on multiple datasets (e.g., UCF-101 and HMDB-51) simultaneously. This acts as a regularizer and allows the network to learn a more generalizable representation by leveraging data from all tasks.

Datasets

  • Training and Evaluation:
    • UCF-101: A benchmark dataset for action recognition with 101 action classes and approximately 13,000 videos.
  • HMDB-51: Another benchmark dataset with 51 action classes and about 6,800 videos.
  • Pre-training:
    • ImageNet: Used to pre-train the spatial stream CNN, leveraging its vast collection of static images.

Key Insights and Ideas

  • Complementarity of Appearance and Motion: The paper strongly demonstrates that spatial (appearance) and temporal (motion) information are complementary for action recognition. Combining them significantly boosts performance beyond what either stream can achieve alone.
  • Optical Flow as a Powerful Motion Representation: Using dense optical flow as input to a CNN for the temporal stream proves highly effective, outperforming raw frame stacks and even hand-crafted features in some contexts. This suggests that explicitly providing motion cues is more beneficial than expecting a network to implicitly learn them from raw pixels.
  • Pre-training is Essential: Pre-training the spatial stream on ImageNet is critical for achieving good performance, especially given the limited size of video datasets. This highlights the transfer learning capabilities of deep CNNs.
  • Multi-task Learning for Data Augmentation: Multi-task learning is an effective strategy to increase the effective training data size and improve generalization, particularly for smaller datasets like HMDB-51.
  • Late Fusion is Sufficient: Simple late fusion of softmax scores from the two streams is effective, avoiding the complexities and potential over-fitting issues of trying to train a single, end-to-end deep network that processes both modalities simultaneously.

Key Results

  • The proposed two-stream ConvNet achieves 87.0% accuracy on the UCF-101 dataset (split 1), which is competitive with state-of-the-art hand-crafted methods.
  • When averaged over three splits, the two-stream model achieves 88.0% accuracy on UCF-101 and 59.4% accuracy on HMDB-51.
  • The temporal stream alone, using optical flow, significantly outperforms the spatial stream alone.
  • The two-stream model outperforms previous deep learning approaches for video classification by a large margin (e.g., "slow fusion" ConvNet [14] achieved 65.4% on UCF-101).
  • Multi-task learning improved the temporal ConvNet's accuracy on HMDB-51 from 46.6% (trained from scratch) to 55.4%.

Limitations and Future Work

  • Camera Motion: While mean subtraction helps, explicit handling of camera motion is still an area for improvement.
  • Local Feature Pooling: The current architecture doesn't explicitly incorporate local feature pooling over spatio-temporal tubes centered at trajectories, a technique used in some successful hand-crafted methods.
  • Large-Scale Datasets: The authors plan to train on larger video datasets (like the one from [14]), which presents significant computational challenges.
  • Architectural Improvements: They suggest that architectures that combine spatial and temporal information more deeply, or integrate ideas from approaches like [30] (deep matching), might further improve performance.

Summary updated

BibTeX
@inproceedings{simonyan2014two,
  author = {Karen Simonyan and Andrew Zisserman},
  title = {{Two-Stream Convolutional Networks for Action Recognition in Videos}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=ryWhCvbd-H},
}
Figure