Learned video codecs compress each frame by predicting it from earlier frames and then transmitting a correction. The best of them now predict the motion itself from motion history, so in much of a frame the transmitted motion correction is exactly zero. The decoder still runs every convolution on every pixel, though. The codec has already identified where nothing new is happening, and that knowledge never reaches the hardware. This note proposes reusing the codec's own motion-update mask as an execution mask. The mask is the small per-region flag, already present in codecs such as ELF-VC, that says "the predicted motion was wrong here, apply a correction". The decoder would gather only the flagged tiles into a dense batch, run its networks on that batch, and fill the remaining tiles by warping cached features from the previous frame.
The sparsity literature explains why this has not happened by default. Zeroing arbitrary activations rarely makes a GPU faster. Fast unstructured kernels need per-matrix preprocessing such as row sorting and permutation, which is affordable for fixed weights but not for patterns that change every frame. A codec avoids this problem because the bitstream already carries the sparsity pattern, at tile granularity and before any computation starts. If that is true in practice, video decoding is a rare workload where dynamic sparsity is free to index. The open questions are whether skipping tiles costs too much rate-distortion quality, and whether errors from skipped tiles accumulate over time.
Core Claim
A predictive neural video decoder can be trained so that a transmitted, tile-aligned motion-update mask directly controls which spatial tiles its motion decoder, context decoder and frame generator compute. At matched rate-distortion quality, measured decoding latency should then fall roughly in proportion to the inactive-tile fraction. This is the claim that separates the idea from prior work: the speedup is measured wall-clock time, not counted multiply-accumulates, and it requires no runtime index construction, because the tile list arrives with the bitstream.
Where This Sits Relative to Prior Work
Two codec papers motivate the idea by showing that motion prediction makes most regions predictable. ELF-VC predicts the current optical flow from previous frames and transmits only a residual plus an update mask, which cuts motion bits. Motion Information Propagation feeds frame context back into motion coding, reducing motion bitrate by about 24%. Neither uses this predictability to skip computation, and the second adds Swin Transformer blocks that make decoding heavier.
The closest codec-side work is masked conditional residual coding (MCRC) from On the Rate-Distortion-Complexity Trade-Offs of Neural Video Coding. MCRC learns a soft spatial mask that switches between conditional and residual coding and reports a 10.08% BD-rate saving. Its mask is continuous and only reweights dense feature maps, so every branch runs everywhere, and complexity is reported only as MACs per pixel. Our mask is binary, tile-aligned and transmitted, and its purpose is to remove work.
On the hardware side, Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity supplies the core execution argument. Zeroing whole spatial positions across all channels removes whole columns of the convolution's matrix form, which yields real speedups of 1.25× to 1.9× on a Raspberry Pi CPU. Its masks are chosen statically or at random for image classification. We replace that choice with a content-driven mask that the decoder receives for free.
Two lines of work constrain the design and rule out the naive alternative. Sparse GPU kernels for deep learning (Sputnik) beats dense cuBLAS only above about 71% sparsity, using sorting and load balancing that assume a fixed pattern. High Performance Unstructured SpMM Computation Using Tensor Cores (SMaT) beats cuBLAS from 78% sparsity, and its authors themselves name the reordering step as a bottleneck for dynamic workloads. Together these imply that thresholding decoder activations per frame and calling an unstructured sparse kernel will not pay off.
MobileNVC shows that block-structured motion compensation on fixed warping hardware reaches real-time 1080p decoding on a phone. We borrow its lesson that motion should be organised in blocks so it maps to hardware, but we apply the block structure to the neural layers rather than only to motion compensation.
Research Program
Start from a DCVC-family conditional codec with MCRC-style temporal prediction at predictor width C=64, the configuration studied in the rate-distortion-complexity paper. Add ELF-VC-style motion extrapolation, so the codec predicts the current flow from the previous two flows. Then quantise the latent and mask computations to a tile grid of 16 to 64 pixels.
The encoder sends one binary flag per tile, entropy-coded with the existing hyperprior. A flagged tile is "active": its motion correction, latent residual and frame-generation layers are computed. An unflagged tile is "skipped": its decoder features at every scale are the previous frame's cached features, warped by the predicted motion. This is the neural counterpart of a classical skip mode.
Convolutions have receptive fields, so active tiles are computed with a halo of neighbouring pixels. Halo pixels read from skipped neighbours take their values from the cache. Execution gathers the active tiles plus halos into a contiguous batch and runs dense convolutions on it, which works with cuDNN on GPU and with XNNPACK-style column skipping on ARM CPUs. No sparse indexing happens inside the kernels.
Training must learn which tiles can be skipped and must handle errors that compound over frames. The flags are trained with a straight-through estimator, in the spirit of the SR-STE estimator from Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch. The training objective is
$$ \mathcal{L} = R + \lambda D + \mu \sum_{i} m_i\, \tau_i , $$
where $R$ is the bitrate, $D$ is the distortion, $\lambda$ sets the rate-distortion trade-off, $m_i \in \{0,1\}$ is the active flag of tile $i$, $\tau_i$ is that tile's measured per-tile latency on the target device (profiled once per architecture), and $\mu$ prices compute. Because $\tau_i$ is measured rather than counted, the objective optimises wall-clock time instead of MACs.
Training uses unrolled sequences longer than Vimeo-90K's seven frames, built by concatenating clips, so the model learns to bound drift from repeatedly reusing cached features. A forced periodic refresh of all tiles serves as a safety net.
The analysis has two parts. The first measures how the active fraction varies across content: videoconference, static camera, pan and high motion. The second checks whether measured speedup tracks the active fraction linearly, which is the signature that indexing overhead has truly disappeared.
Baselines
The naive reference point is the same codec decoding densely: an identical MCRC model at C=64, trained on the same data for the same number of steps, with flags forced to one. Every speedup and BD-rate figure is reported relative to it. The rate-distortion-complexity paper reports a 10.08% BD-rate saving for MCRC at C=64 over conditional coding at C=64, and we expect our dense retraining to reproduce that ordering. The proposal beats this baseline if, within 1% BD-rate of it, decoding is at least 1.5× faster at 1080p on HEVC Class E and UVG with a static camera.
The strongest cheap complexity baseline is width reduction. Conditional residual coding (CRC) at C=32 comes from the same paper, which reports that it matches conditional coding at C=64 with about 22% fewer decoding MACs. CRC is the strongest competitor because it needs no masks and no gather step. We retrain it on our data and pipeline with the same tuning budget, measure its wall-clock speed on the same devices, and require our method to reach a better point on the BD-rate versus measured-latency curve.
Slimmable Video Codec is reported as a reference for aggressive slimming. Its 20× FLOP reduction comes with rate-distortion only comparable to H.264 veryfast, so it bounds the low-quality end of the curve.
The strongest rate-distortion method in the file is Motion Information Propagation, which reports a 12.9% bitrate reduction over DCVC-HEM and a 16.6% PSNR-measured gain over VTM-13.2. We do not expect to beat it on rate-distortion. It serves as the quality ceiling, and we report our gap to it alongside latency. No reported decoding latency for it was found on Lacuna, so we profile it ourselves on the same GPU.
MobileNVC reports 38.9 FPS 1080p decoding on a Snapdragon 8 Gen 2 and 48% BD-rate savings over MobileCodec. We cannot run on its warping hardware, so we cite it for context only and do not claim an FPS comparison.
Two ablations isolate the key ingredient, which is that the mask comes from motion prediction and arrives in the bitstream. The first replaces the transmitted mask with a random mask of equal density, following the semi-structured activation sparsity paper's training recipe. It tests whether the choice of tiles matters. The second replaces the transmitted mask with per-frame thresholding of decoder activations executed through Sputnik, including the time to build its sparse format. It tests whether bitstream-supplied indexing matters. Further ablations vary tile size and halo width, and compare hard masks against MCRC's soft mask at equal latency.
Why not just…?
Why not just narrow the channels, as CRC at C=32 does? Width reduction is portable and is the real threat in this comparison. However, its savings are uniform: a near-static videoconference frame costs as much as a frenetic one. The rate-distortion-complexity paper also shows that narrowing conditional coding inflates BD-rate by about 9–10%. Our method's advantage should grow with how static the content is. If CRC at C=32 matches us on the measured-latency curve for HEVC Class E, the idea fails, and the experiment is designed to show this directly. The two methods also combine naturally, and we test CRC at C=32 with tile masks.
Why not just apply 2:4 weight sparsity on Ampere tensor cores? N:M weight sparsity gives at most 2× on supported hardware, according to the N:M paper, and gives nothing on older GPUs or edge CPUs. It is also orthogonal to our method: it removes work inside every tile, while we remove whole tiles. We will report the combination. If we report only one, it will be ours, because it runs on any device that supports dense convolution.
Why not just threshold activations and use a fast unstructured sparse kernel? Sputnik and SMaT break even with dense cuBLAS only at 71–78% sparsity, and both depend on preprocessing that SMaT's authors flag as a problem for dynamic data. Our second ablation measures this directly, with format-building time included. We expect it to lose to dense decoding at realistic per-frame sparsity.
Why not just use block motion compensation on dedicated hardware, as MobileNVC does? That approach speeds up warping, but every neural layer still runs densely on the NPU. Our mask targets exactly those layers, and the two approaches stack.
Proposed Experiments and Decisive Checks
Training uses Vimeo-90K with concatenated long sequences. Testing uses UVG, HEVC Classes B, C and E, and MCL-JCV. Metrics are BD-rate in PSNR and MS-SSIM against VTM and against the dense reference, active-tile fraction per sequence, and wall-clock decoding FPS at 1080p on an A100 and a V100 GPU and on an ARM CPU. Latency is reported as median and 99th percentile, including the cost of gathering tiles and warping the cache.
Three outcomes decide the idea. The first is whether measured speedup follows $1/(\text{active fraction} + \text{halo overhead})$ closely. A large shortfall would mean the gather step reintroduces the overhead the idea is meant to remove. The second is whether, within 1% BD-rate of dense decoding, Class E and other static-camera content reach at least 1.5× speedup while high-motion UVG sequences degrade gracefully towards 1×. The third is whether the drift added by cached features stays under 0.2 dB PSNR across a 96-frame group of pictures. If the first check fails, the hardware premise is wrong. If the third fails, the skip mode needs a learned correction, and a weaker version of the claim remains: a decoder-side mask with a residual refresh.
References
- ELF-VC: Efficient Learned Flexible-Rate Video Coding
- Motion Information Propagation for Neural Video Compression
- On the Rate-Distortion-Complexity Trade-Offs of Neural Video Coding
- Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity
- Sparse GPU Kernels for Deep Learning
- High Performance Unstructured SpMM Computation Using Tensor Cores
- MobileNVC: Real-time 1080p Neural Video Compression on a Mobile Device
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- Slimmable Video Codec