Delving Deeper into Convolutional Networks for Learning Video Representations Nicolas Ballas

Li Yao, Pal Chris, Aaron Courville

Introduction

Standard approaches for video analysis typically rely on a "two-step" process: extract high-level features from the final layer of a 2D Convolutional Neural Network (CNN) and feed those features into a Recurrent Neural Network (RNN) to model temporal changes. However, this method discards crucial fine-grained spatial information.

This paper introduces the GRU-RCN (Gated Recurrent Unit - Recurrent Convolutional Network), a novel architecture that captures fine-grained video motion by applying GRUs to intermediate layers of a CNN. By replacing fully connected layers in the GRU with convolutions, the model preserves spatial topology and learns spatio-temporal features across multiple resolutions, significantly improving action recognition and video captioning.

The Problem: Spatial Invariance vs. Local Motion

CNNs are inherently designed to be spatially invariant. As data moves deeper into a CNN, the network discards local spatial information to focus on global object categories. While excellent for image classification, this is a major flaw for video analysis.

In video, motion is often local and smooth—consider the subtle movement of a hand versus a sudden camera cut. By only looking at the "top" layer of a CNN, models miss these fine-grained motion patterns. The challenge lies in leveraging the rich, high-resolution information present in the intermediate layers of a CNN without causing an explosion in the number of parameters.

The Solution: Recurrent Convolutional Networks (RCN)

To address this, the authors propose the Recurrent Convolutional Network, specifically utilizing a variant of the Gated Recurrent Unit (GRU).

Multi-resolution feature maps extracted over a temporal sequence of video frames.

The architecture is built on three core principles:

  1. Multi-Level Visual Percepts: Instead of relying solely on the final layer, the model extracts features (termed "percepts") from multiple levels of a pre-trained CNN like VGG-16 or GoogleNet (e.g., from pool2 through fc7).
  2. Convolutional GRU (GRU-RCN): In a standard GRU, transitions are fully connected. This ignores spatial structure and requires a massive number of parameters for high-resolution maps. The authors replace these linear operations with convolutions. This enforces sparse connectivity and parameter sharing, allowing the model to process 3D tensors (spatial dimensions plus channels) while maintaining the frame's spatial topology.
  3. Stacked & Bi-directional Variants:
    • Stacked GRU-RCN: Adds bottom-up connections between RCN layers, allowing the model to integrate information across different resolutions.
  4. Bi-directional GRU-RCN: Processes the video sequence in both forward and backward temporal orders to capture a more complete context.

Key Insights

  • Locality Matters: Video motion is restricted to local neighborhoods. By using convolutional units in the RNN, the model inherently respects this "locality prior," making it much more efficient at capturing motion than a fully connected RNN.
  • The Resolution Trade-off: High-level percepts are discriminative but blurry, whereas low-level percepts are spatially sharp but lack semantic meaning. The GRU-RCN bridges this gap by modeling temporal dynamics at every level of the hierarchy.
  • Efficiency through Sparsity: Replacing fully connected layers with $3 \times 3$ convolutions drastically reduces the parameter count. For a typical feature map, a fully connected RNN would require $O(N^2)$ parameters, while the convolutional version requires only $O(k^2)$, where $k$ is the small kernel size.

Experimental Results

The model was evaluated on two primary benchmarks: UCF101 (human action recognition) and YouTube2Text (video captioning).

  • Action Recognition: On UCF101 (split 1), the Bi-directional GRU-RCN achieved 80.7% accuracy using only RGB inputs, outperforming a standard VGG-16 (78.0%) and a VGG-16 + RNN baseline (78.1%). When combined with optical flow, the model reached 90.8%, rivaling state-of-the-art methods of the time.
  • Video Captioning: On YouTube2Text, the model demonstrated massive improvements. A GoogleNet + Bi-directional GRU-RCN reached a BLEU score of 0.4963, significantly higher than the VGG-16 baseline (0.3700).
  • Simplicity Wins: Notably, the authors achieved these results using a simpler caption-decoder and without the need for computationally expensive 3D CNN features (like C3D).

Limitations and Future Work

Despite its success, the approach has a few limitations:

  • Data Hunger: The authors noted that the Stacked GRU-RCN actually performed slightly worse than the standard GRU-RCN on UCF101. They attribute this to the difficulty of training deeper recurrent structures on relatively small datasets (the dataset has only 9,500 training videos).
  • Pre-trained Dependence: The model relies heavily on a CNN pre-trained on ImageNet. While effective, end-to-end training of the entire spatio-temporal stack could potentially unlock even better representations.
  • Computational Cost: While more parameter-efficient than fully connected RNNs, running multiple convolutional GRUs across many layers still adds significant overhead during inference compared to single-layer approaches.

Summary updated

Abstract

We propose an approach to learn spatio-temporal features in videos from intermediate visual representations we call "percepts" using Gated-Recurrent-Unit Recurrent Networks (GRUs).Our method relies on percepts that are extracted from all level of a deep convolutional network trained on the large ImageNet dataset. While high-level percepts contain highly discriminative information, they tend to have a low-spatial resolution. Low-level percepts, on the other hand, preserve a higher spatial resolution from which we can model finer motion patterns. Using low-level percepts can leads to high-dimensionality video representations. To mitigate this effect and control the model number of parameters, we introduce a variant of the GRU model that leverages the convolution operations to enforce sparse connectivity of the model units and share parameters across the input spatial locations. We empirically validate our approach on both Human Action Recognition and Video Captioning tasks. In particular, we achieve results equivalent to state-of-art on the YouTube2Text dataset using a simpler text-decoder model and without extra 3D CNN features.
BibTeX
@inproceedings{yao2016delving,
  author = {Li Yao and Pal Chris and Aaron Courville},
  title = {{Delving Deeper into Convolutional Networks for Learning Video Representations Nicolas Ballas}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1511.06432},
}
Figure