Abstract
We propose Skip-Convolutions to leverage the large amount of redundancies in video streams and save computations. Each video is represented as a series of changes across frames and network activations, denoted as residuals. We reformulate standard convolution to be efficiently computed on residual frames: each layer is coupled with a binary gate deciding whether a residual is important to the model prediction, e.g. foreground regions, or it can be safely skipped, e.g. background regions. These gates can either be implemented as an efficient network trained jointly with convolution kernels, or can simply skip the residuals based on their magnitude. Gating functions can also incorporate block-wise sparsity structures, as required for efficient implementation on hardware platforms. By replacing all convolutions with Skip-Convolutions in two state-of-the-art architectures, namely EfficientDet and HRNet, we reduce their computational cost consistently by a factor of 3-4x for two different tasks, without any accuracy drop. Extensive comparisons with existing model compression, as well as image and video efficiency methods demonstrate that Skip-Convolutions set a new state-of-the-art by effectively exploiting the temporal redundancies in videos.
Introduction
Standard deep learning models typically treat video as a sequence of independent, static images. This approach is inherently inefficient; even in a video where a subject remains stationary against a fixed background, a traditional Convolutional Neural Network (CNN) re-calculates every pixel and feature map for every single frame.
While video compression standards like HEVC have long exploited the fact that most pixels do not change significantly between frames, deep learning architectures have struggled to implement similar efficiencies. Previous attempts at "conditional computation"—skipping parts of an image—often fail because identifying "unimportant" regions in a static image is non-trivial. This research argues that in video, the "residual" (the difference between consecutive frames) provides a natural roadmap for where a network should focus its computational resources.
The Skip-Convolution Framework
The core contribution of this work is the Skip-Convolution, which reformulates standard convolutions to operate on changes rather than full frames. This framework relies on three primary components:
- Residual Reformulation: Leveraging the linearity of convolution, the output of a current frame $z_t$ can be calculated as the output of the previous frame $z_{t-1}$ plus the convolution of the residual (the difference between the current and previous frames). Mathematically: $z_t = z_{t-1} + w * (x_t - x_{t-1})$.
- Gating Functions: To achieve efficiency, the model avoids computing the convolution on the residual if the change is negligible. The authors propose two gating strategies:
- Norm Gates: A simple thresholding method based on the residual's magnitude that requires no additional training.
- Gumbel Gates: A lightweight, trainable sub-network that learns to identify which changes are task-relevant (e.g., ignoring swaying trees while focusing on a moving vehicle).
- Structured Sparsity: To ensure hardware compatibility, gates operate on blocks (e.g., 4x4 or 8x8 pixels) rather than individual pixels, aligning with the processing patterns of modern GPUs and AI accelerators.
Experimental Evaluation
The researchers evaluated Skip-Convolutions across two distinct computer vision tasks: Object Detection using the UA-DETRAC dataset (140,000+ frames of traffic video) and Human Pose Estimation using the JHMDB dataset.
Key Results
- EfficientDet (Object Detection): Skip-Convolutions reduced computational cost (MACs) by 300% to 350% across various configurations (D0-D3) with no loss in Average Precision.
- HRNet (Pose Estimation): The method achieved a 2× reduction in MACs while simultaneously increasing the accuracy (PCK score) compared to the baseline model.
- Real-world Performance: On standard CPUs, theoretical MAC reductions translated to a ~2× wall-clock speedup. The authors suggest that specialized sparse kernels could unlock even greater performance gains.
Insights and Future Directions
A significant finding of this work is that residuals serve as a "strong prior." Identifying important regions in a static image is difficult, but in a residual frame, regions that haven't changed are almost certainly safe to skip. Furthermore, the Gumbel gates appeared to have a regularizing effect, sometimes improving accuracy by preventing overfitting through their stochastic nature during training.
Skip-Convolutions are also complementary to existing model compression techniques. Because they exploit temporal redundancy rather than spatial or channel-wise redundancy, they can be combined with methods like pruning or quantization for multiplicative efficiency gains.
Limitations
The primary challenge remains significant camera motion. Rapid panning or shaking can create "noisy" residuals, potentially causing the gating mechanism to bypass fewer regions. Future work aims to integrate learnable warping functions (optical flow) to align frames before calculating residuals, which would restore sparsity and efficiency even in dynamic filming conditions.
Summary updated