Delving Deeper into Convolutional Networks for Learning Video Representations.

Nicolas Ballas, Li Yao, Chris Pal, Aaron C. Courville

TL;DR

This paper introduces a novel Recurrent Convolutional Network (RCN) architecture that leverages intermediate feature maps from deep convolutional neural networks (CNNs) to better capture fine-grained motion patterns in videos. By applying Gated Recurrent Units (GRUs) to these lower-level, higher-resolution "percepts," the authors achieve state-of-the-art results on video captioning and strong performance on action recognition tasks.

What Problem Does This Paper Address?

Understanding video content is a significant challenge in computer vision. Traditional approaches relied on hand-crafted features, but the trend is towards learning general video representations. While 2D CNNs excel at image tasks and Recurrent Neural Networks (RNNs) handle temporal sequences, combining them for video (Recurrent Convolutional Networks or RCNs) has shown promise. However, existing RCNs typically use only the highest-level, most abstract features from CNNs. These features, while discriminative, lose spatial detail. The authors argue that this loss of spatial resolution hinders the ability to model fine motion patterns, which are crucial for video understanding, especially for tasks requiring detailed temporal analysis. They aim to bridge this gap by utilizing features from multiple layers of a CNN, thereby capturing both discriminative power and fine spatial detail.

Methods

The core innovation lies in a new RCN architecture that applies GRUs to intermediate convolutional feature maps extracted from a pre-trained CNN (like VGG-16).

  • Leveraging Intermediate Percepts: Instead of just using the final layer's output, the authors extract "percepts" (feature maps) from multiple convolutional layers. Lower layers offer higher spatial resolution, preserving motion details, while higher layers offer more semantic information.
  • Convolutional GRU-RCN: To handle the high dimensionality of these intermediate features and avoid a combinatorial explosion of parameters, they introduce a convolutional GRU. This means the GRU's internal operations (like the update and reset gates) are performed using convolutions instead of fully connected layers. This enforces sparsity and parameter sharing across spatial locations, making the model more efficient and better suited to the grid-like structure of feature maps.
  • Stacked GRU-RCN: They also propose a "stacked" version where GRU-RCN layers are connected in a bottom-up fashion, allowing information to flow between different levels of abstraction.
  • Bi-directional GRU-RCN: To capture temporal information more comprehensively, they explore a bi-directional GRU-RCN that processes sequences in both forward and reverse temporal orders.

Datasets

  • Human Action Recognition: UCF101 dataset.
  • Video Captioning: YouTube2Text dataset.

The paper does not introduce new datasets but evaluates on existing, well-established benchmarks.

Key Insights and Ideas

  • The Importance of "Deeper" Percepts: The key insight is that high-level CNN features, while semantically rich, discard too much spatial information for fine motion modeling. Lower-level, higher-resolution feature maps are crucial for capturing subtle temporal dynamics.
  • Convolutional GRUs for Efficiency and Structure: Applying standard GRUs to convolutional maps would be computationally prohibitive. Using convolutional GRUs elegantly addresses this by leveraging the spatial structure of the feature maps and sharing parameters, leading to a more efficient and effective model.
  • Bridging Spatial and Temporal: The proposed RCN architecture effectively bridges the gap between the spatial feature extraction capabilities of CNNs and the temporal modeling power of RNNs, by operating on features that retain both spatial detail and semantic richness.

Key Results

  • Action Recognition: The proposed GRU-RCN variants outperform baseline VGG-16 and VGG-16 RNN models by a significant margin (e.g., 3.4% gain for Bi-directional GRU-RCN over baselines). The Bi-directional GRU-RCN achieved the best performance among their GRU-RCN variants (80.7% accuracy).
  • Video Captioning: The approach achieves state-of-the-art results on the YouTube2Text dataset. Specifically, their GoogleNet + Bi-directional GRU-RCN encoder, combined with a standard decoder, significantly outperforms previous methods, achieving a 10% gain in BLEU score over a VGG-16 encoder baseline and matching other state-of-the-art results without requiring more complex 3D CNN encoders.

Limitations and Future Work

  • Stacked GRU-RCN Performance: The Stacked GRU-RCN showed lower performance, which the authors attribute to the limited training data in UCF101 making deeper models harder to train.
  • Computational Cost: While more efficient than fully-connected RNNs on convolutional maps, the RCNs still involve significant computation.
  • Future Work: The paper implicitly suggests exploring more sophisticated ways to combine features from different layers, potentially with attention mechanisms, and investigating larger datasets to better train deeper RCN architectures. The success on video captioning also hints at further exploration of encoder-decoder architectures for video understanding tasks.

Summary updated

Abstract

We propose an approach to learn spatio-temporal features in videos from intermediate visual representations we call "percepts" using Gated-Recurrent-Unit Recurrent Networks (GRUs).Our method relies on percepts that are extracted from all level of a deep convolutional network trained on the large ImageNet dataset. While high-level percepts contain highly discriminative information, they tend to have a low-spatial resolution. Low-level percepts, on the other hand, preserve a higher spatial resolution from which we can model finer motion patterns. Using low-level percepts can leads to high-dimensionality video representations. To mitigate this effect and control the model number of parameters, we introduce a variant of the GRU model that leverages the convolution operations to enforce sparse connectivity of the model units and share parameters across the input spatial locations. We empirically validate our approach on both Human Action Recognition and Video Captioning tasks. In particular, we achieve results equivalent to state-of-art on the YouTube2Text dataset using a simpler text-decoder model and without extra 3D CNN features.
BibTeX
@inproceedings{ballas2016delving,
  author = {Nicolas Ballas and Li Yao and Chris Pal and Aaron C. Courville},
  title = {{Delving Deeper into Convolutional Networks for Learning Video Representations.}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1511.06432},
}
Figure