When a convolutional network runs on video, most of its work is repeated. Consecutive frames look almost the same, so many internal feature maps come out almost the same as they did a fraction of a second earlier. Separately, filter-pruning research has shown that many filters inside a layer duplicate one another, so they can be removed with little loss. This proposal joins the two observations at the level of individual filters. For each output channel of each layer, we ask two questions. First, can its output be reconstructed from the other channels in the same frame? If so, the filter is redundant across channels and is pruned permanently. Second, can its output be predicted from its own value a few frames ago? If so, it is redundant over time, so it is computed only every few frames and reused in between. Every remaining filter runs on every frame.
The bet is that redundancy over time is concentrated in specific filters and stays put. Some filters respond to slow scene properties such as road, sky and texture, and others respond to edges in motion. If that holds, a network can be given a per-channel update rate, which works like a learned, fine-grained version of the hand-designed slow and fast pathways of SlowFast. Because the skipped computation is a contiguous block of output channels rather than scattered pixels, the savings should show up as real latency on ordinary hardware, not only as fewer theoretical operations.
Core Claim
For a trained video convolutional network, output channels can be split into three groups using statistics measured on video: prunable, cacheable and live. Running the cacheable channels at a reduced rate, and training the network to tolerate the resulting stale inputs, gives a better trade between accuracy and measured frames per second than any of the following at the same compute: static similarity pruning alone, spatial feature reuse alone, or whole-network keyframe reuse.
Concretely, let $a_c(t)$ be the activation map of channel $c$ at frame $t$. We use two scores. The first is cross-channel redundancy $r_c$, the fraction of $a_c(t)$'s energy left unexplained after a least-squares reconstruction from the other retained channels in the same frame. This is the quantity that Principal Filter Analysis and the cosine-similarity criterion of Feature Statistics Guided Efficient Filter Pruning approximate on still images. The second is temporal staleness at lag $k$:
$$s_c(k) = \frac{\mathbb{E}_t \|a_c(t+k) - a_c(t)\|^2}{\mathbb{E}_t \|a_c(t)\|^2},$$
the relative error from reusing a channel's output $k$ frames later. A channel with small $r_c$ is pruned. A channel with small $s_c(k)$ is assigned update period $k$. The pruning criterion is not new. What is new is the second axis, and treating the two axes jointly under a single compute budget.
Relation to Prior Work
The work closest to ours uses temporal similarity but at a coarser grain. Distortion-Aware Network Pruning and Feature Reuse (STMG) does three things:
- it uses cosine similarity between consecutive frames' features to reuse static spatial regions;
- it uses a Bayesian gate to drop whole residual blocks on low-change frames;
- it reaches 32.6 frames per second, up from 22.1, on DDRNet-39 on Cityscapes, with mIoU falling from 80.3% to 79.1%.
STMG never asks which filters are stable. It reuses all channels at a location or none of them, and it skips all channels of a block or none. Our method reuses some channels everywhere and keeps the rest live everywhere, so the two can in principle be stacked.
Residual Sparsity Connection Learning for Video Super-Resolution does prune channels in a temporal model. It ranks them by the size of learned scaling factors, and its pruning is static. We reuse its key insight: errors compound across time steps unless training aligns the pruned model's states with a teacher over long sequences. That insight shapes our stale-input fine-tuning.
The static similarity pruners supply our cross-channel criterion and serve as baselines. These include FPGM, which ranks filters by closeness to the layer's geometric median in weight space, C-SGD, which pulls clustered filters together during training until duplicates can be removed, PFA, and DFS/SFS. All of them measure redundancy within a single image, so none can tell a background-texture channel from a motion channel.
Learning Disentangled Representation in Pruning for UAV Tracking prunes a video tracker by feature-map rank. It still treats each frame independently.
Token methods for video transformers attack a different axis, the number of tokens rather than channel width. STTS and DyCoke drop or merge redundant spatiotemporal tokens. STTS's finding that dropping tokens breaks the spatial grids that convolutions need is part of why we work on channels, which leave the grid intact.
Research Program
Phase 1: measurement. The whole proposal depends on an empirical fact that may be false. Temporal staleness has to be spread unevenly across channels, and a channel's staleness has to be consistent across videos and not just across frames of one clip. On a pretrained DDRNet-39 on Cityscapes video sequences, and on a ResNet-50 SlowFast on Kinetics-400 and Something-Something-v2, we compute $s_c(k)$ for $k$ from 1 to 8 for every channel.
We then check two things. First, whether the distribution of staleness within a layer is wide or bimodal. Second, whether the ranking of channels by staleness is stable across held-out videos, measured by rank correlation between calibration and test splits. We also measure whether $r_c$ and $s_c$ are correlated. If they are strongly correlated, temporal caching adds little beyond pruning. If they are weakly correlated, the axes are complementary, much as DFS/SFS found its two image-level statistics correlated at only −0.38.
If staleness turns out to be spread evenly, or to depend on the input, the caching idea fails. In that case we report the negative result together with its per-layer evidence.
Phase 2: method. Within each layer we reorder channels so that live and cached output channels are contiguous. On a non-refresh frame, the convolution computes only the live slice, and the stale cached slice is concatenated from a buffer. Compute for that layer falls in proportion to the cached fraction.
We assign update periods by a greedy search under a global budget on average operations per frame. This mirrors the global, rather than layer-local, allocation that the video super-resolution work found superior.
We then fine-tune with stale inputs. During training, cached channels are fed values from $k$ frames earlier, and a distillation loss aligns the outputs with the unpruned teacher over multi-frame windows. This is the analogue of temporal fine-tuning in the video super-resolution work, and it is what should keep accumulated error, or drift, bounded without frequent full refreshes.
Phase 3: clip models. For clip-level recognition, the same assignment becomes a per-channel temporal stride. A cached channel is computed at $T/k$ time positions and repeated in between. This lets us test directly whether the data-driven per-channel rates beat SlowFast's two fixed rates.
Baselines
The naive reference points are the unmodified network on every frame, which sets the accuracy ceiling and the speed floor, and whole-network keyframe reuse. In keyframe reuse, every feature is recomputed every $k$ frames, the output is reused in between, and $k$ is chosen to match our average compute. This is the cheapest possible use of temporal redundancy, and on slowly changing driving scenes it may be hard to beat.
The strongest existing temporal method is STMG on DDRNet-39 on Cityscapes. The paper reports 79.1% mIoU at 32.6 frames per second against 80.3% at 22.1 for the full model. We will rerun STMG on the same GPU and the same frame sequences so that frame rates are comparable. Beating it means higher mIoU at equal measured frame rate, or higher frame rate at equal or better mIoU. A gain inside run-to-run variance across three seeds does not count.
The strongest static competitor is the best of FPGM and PFA. Both will be applied to the same backbone, with similarity statistics computed on video frames rather than still images, pruned to the same average operations per frame, and fine-tuned with the same epoch budget and distillation teacher that our method gets. For PFA on VGG-16 on CIFAR-10, the reported figure is an 8× reduction in trainable variables with a 0.4% accuracy gain. No reported result was found on Lacuna for FPGM or PFA on Cityscapes video segmentation, so our runs will establish that baseline.
On recognition, the reference is ResNet-101 SlowFast at 79.8% top-1 on Kinetics-400. STTS on MViT is reported to keep accuracy within 0.7% while cutting 33% of operations on Kinetics-400 and SSv2. It uses a different backbone, so it will be cited as context rather than matched head to head.
Three ablations isolate the key ingredient:
- Random assignment. The same fraction of channels is cached, chosen at random. This tests whether the staleness score matters at all.
- Magnitude assignment. Channels are cached by weight norm instead.
- No stale-input fine-tuning. The full method is run without the training step that exposes the network to stale inputs.
We also stack our channel caching on top of STMG's spatial reuse, to test whether the two are complementary.
Why not just…?
Why not just run PFA or FPGM on video frames and stop? This is the most natural extension and a real competitor. If most temporally stale channels are also redundant across channels, it will match us. Phase 1's correlation between $r_c$ and $s_c$ predicts which outcome to expect. The comparison at matched operations per frame then settles it. We claim a win only if pruning plus caching beats pruning alone at the same budget. Pruning cannot exploit redundancy over time by construction, because a filter that captures a unique but slowly changing feature is irreplaceable within a frame yet wasted on most frames.
Why not just reuse static spatial regions as STMG does? Spatial reuse saves compute only where the whole scene is still. Its masks produce irregular, non-contiguous computation, and the deployment literature notes that such computation often fails to turn into latency on edge hardware. Channel caching saves compute even when the camera moves, provided the moving content is captured by a minority of channels. The stacked experiment shows whether the two are substitutes or complements. If STMG alone matches stacked performance, we will say so.
Why not just recompute everything on keyframes and interpolate or reuse in between? On slowly varying driving video this baseline may be strong. It is included at matched compute for exactly that reason. Its predicted weakness is fast-changing content, where reusing all channels discards motion that a few live channels would have kept. SSv2 is the stress test: SlowFast and UniFormer both show that temporal resolution matters there. Our method should degrade much more gracefully than keyframe reuse as $k$ grows. If it does not, the per-channel premise is wrong.
Decisive Checks
Three results decide the proposal. The first is Phase 1: staleness must be concentrated in a subset of channels and stable across videos (high rank correlation between calibration and test splits). Otherwise the method has no basis. The second is the matched-budget comparison against the best video-calibrated static pruner on Cityscapes, plus the random-assignment ablation. Together these show whether the temporal axis adds anything beyond cross-channel similarity. The third is measured latency against STMG on the same GPU, reported alongside activation-buffer memory. Cached channels cost memory, and the gain only counts if it survives that accounting.
As a secondary, interpretive check, we will see whether the learned per-channel rates on SlowFast recover a slow/fast split by themselves. We will also see whether they cache more of the fast pathway than SlowFast's fixed design implies, or less.
References
- SlowFast Networks for Video Recognition
- Filter Distillation for Network Compression
- Feature Statistics Guided Efficient Filter Pruning
- Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation
- Residual Sparsity Connection Learning for Efficient Video Super-Resolution
- Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
- Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
- Learning Disentangled Representation in Pruning for Real-Time UAV Tracking
- Efficient Video Transformers with Spatial-Temporal Token Selection
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models