Skip-Convolutions for Efficient Video Processing

Amirhossein Habibian, Davide Abati, Taco S. Cohen, Babak Ehteshami Bejnordi

TL;DR

This paper introduces "Skip-Convolutions" (Skip-Convs) to significantly speed up video processing by focusing computations only on the changes between frames (residuals). By using learned "gates" to decide which residual information is important, Skip-Convs reduce computational cost by 3-4x in state-of-the-art architectures without sacrificing accuracy, setting a new benchmark for efficient video analysis.

What Problem Does This Paper Address?

Processing video data with deep neural networks is computationally expensive. Standard approaches treat each video frame as an independent image, leading to redundant computations, especially when consecutive frames are very similar. This inefficiency limits the deployment of complex, accurate models for real-time applications like object detection or pose estimation on video streams. The challenge is to leverage the inherent temporal redundancy in videos to reduce computational load without compromising the model's performance.

Methods

The core idea is to reformulate standard convolutions to operate on residual frames rather than full frames. A residual frame represents the difference between the current frame and the previous one. The authors propose "Skip-Convolutions" where each convolutional layer is augmented with a gating function. This gate, acting on the residual frame, decides which parts of the residual are important enough to warrant computation.

  • Residual Computation: Instead of computing Zt = W * Xt, they compute Zt = Zt-1 + W * rt, where rt = Xt - Xt-1 is the residual. This means only the changes are processed.
  • Gating Functions: To further optimize, a binary gate g(rt) is introduced. This gate predicts a mask, indicating whether to compute the convolution on a residual region or simply copy the feature from the previous time step (Zt-1).
  • Norm Gate: A simple, non-trainable gate that skips residuals based on their magnitude (e.g., if the residual is small). This can be plugged into existing models without retraining.
  • Gumbel Gate: A trainable gate that is learned jointly with the network. This allows for more sophisticated skipping decisions, potentially skipping even larger residuals if they are deemed unimportant for the task.
  • Structured Sparsity: To facilitate efficient hardware implementation, the gates can be structured into blocks (e.g., 8x8), which aligns with how many hardware accelerators process data.

Datasets

The paper evaluates its method on two primary datasets:

  • UA-DETRAC: A large-scale dataset for vehicle detection in traffic videos.
  • JHMDB: A dataset for human pose estimation in videos.

They also mention preliminary experiments on video classification, using a dataset for human action recognition.

Key Insights and Ideas

  • Residuals as a Strong Prior: The key insight is that residual frames are inherently sparse and contain strong cues about important changes, making them ideal for conditional computation. This is more effective than applying similar techniques to raw image frames.
  • Temporal Redundancy is Underutilized: Existing efficient video models often rely on feature propagation at a single layer or complex alignment mechanisms. Skip-Convs offer a more general approach by applying efficient residual processing at every layer.
  • Gating is Crucial for Efficiency: The learned gating mechanism is vital for distinguishing truly important changes from noise or minor variations, allowing for significant computational savings.
  • Complementary to Model Compression: Skip-Convs tackle temporal redundancy, while traditional model compression methods address spatial or channel redundancies. These can be combined for even greater efficiency gains.

Key Results

  • Significant Speedup: Skip-Convs consistently reduce computational cost (measured in GMACs) by a factor of 3-4x in state-of-the-art architectures like EfficientDet and HRNet.
  • No Accuracy Drop: Crucially, these speedups are achieved without any significant drop in accuracy on both object detection and pose estimation tasks.
  • State-of-the-Art Efficiency: For object detection on UA-DETRAC, Skip-Conv combined with EfficientDet achieves state-of-the-art accuracy with orders of magnitude fewer computations compared to previous methods.
  • Improved Pose Estimation: On JHMDB, Skip-Convs improve the PCK metric while reducing MACs by about 2x compared to the baseline HRNet.
  • Structured Gating Benefits: Structured gating (e.g., 4x4 blocks) provides hardware-friendly sparsity without negatively impacting accuracy or efficiency.

Limitations and Future Work

  • Severe Camera Motion: The authors acknowledge that in scenarios with very aggressive camera motion, residual frames might contain less useful information, potentially putting a higher burden on the gating function.
  • Future Work: They suggest exploring learnable warping functions to compensate for severe camera motions and generalizing Skip-Convs to a broader set of transformations beyond simple subtraction/addition.

Summary updated

Abstract

We propose Skip-Convolutions to leverage the large amount of redundancies in video streams and save computations. Each video is represented as a series of changes across frames and network activations, denoted as residuals. We reformulate standard convolution to be efficiently computed on residual frames: each layer is coupled with a binary gate deciding whether a residual is important to the model prediction,~\eg foreground regions, or it can be safely skipped, e.g. background regions. These gates can either be implemented as an efficient network trained jointly with convolution kernels, or can simply skip the residuals based on their magnitude. Gating functions can also incorporate block-wise sparsity structures, as required for efficient implementation on hardware platforms. By replacing all convolutions with Skip-Convolutions in two state-of-the-art architectures, namely EfficientDet and HRNet, we reduce their computational cost consistently by a factor of 3~4x for two different tasks, without any accuracy drop. Extensive comparisons with existing model compression, as well as image and video efficiency methods demonstrate that Skip-Convolutions set a new state-of-the-art by effectively exploiting the temporal redundancies in videos.
BibTeX
@inproceedings{habibian2021skip,
  author = {Amirhossein Habibian and Davide Abati and Taco S. Cohen and Babak Ehteshami Bejnordi},
  title = {{Skip-Convolutions for Efficient Video Processing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=rfGbMit_sl5},
}
Figure