TL;DR
This paper introduces RDOSkip, a rate-distortion optimized skip mode framework for neural video compression (NVC). By replacing rigid, fixed-threshold skip decisions with an adaptive method that accounts for spatial and temporal distortion variations, the authors achieve significant bitrate savings (up to 1.29% BD-rate) with zero retraining and negligible computational overhead.
What Problem Does This Paper Address?
In video compression, "skip mode" is a powerful tool: if a part of a frame can be accurately predicted from previous frames, we simply "skip" encoding the residual (the difference) to save bits. In traditional codecs like HEVC, this decision is made using rigorous Rate-Distortion Optimization (RDO).
However, in Neural Video Compression (NVC), skip mode has been handled much more crudely. Most current NVC frameworks use a fixed threshold based on the variance predicted by a hyperprior. If the predicted variance is low, the system assumes the prediction is "good enough" and skips it.
The authors identify a major flaw here: variance does not equal distortion. As shown in the figure below, feature elements with the same variance can result in wildly different Mean Squared Error (MSE) in the final reconstructed image. Furthermore, the sensitivity to these errors changes depending on which feature channel you are in and the quality of the reference frame used for prediction.
Methods
The researchers propose a framework that bridges the gap between feature-domain decisions and pixel-domain results.
- Bayesian Distortion Formulation: They use Bayes' theorem to derive a mathematical link between the loss in the latent feature domain ($|\Delta y|$) and the resulting MSE in the reconstructed pixels. This allows the encoder to estimate the "cost" of skipping a feature without actually running the full decoder.
- Spatially Adaptive Thresholds: Instead of one threshold for the whole frame, RDOSkip calculates channel-wise thresholds. It uses the first and second-order statistics (mean and standard deviation) of the predicted variances within a channel to proxy the "Jacobian matrix" of the decoder—essentially measuring how sensitive a specific channel is to noise.
- Temporally Adaptive Thresholds: The quality of a video frame in NVC often depends on its position in a Group of Pictures (GOP). RDOSkip adjusts the skip threshold based on the Lagrange multiplier ($\lambda$) of the current frame and its reference frame, accounting for the fact that low-quality references make residuals harder to skip.
- Inference-Only Integration: Crucially, this method is "plug-and-play." It requires no retraining of the underlying neural networks and can be directly applied to existing NVC models during inference.
Datasets
The authors evaluated their framework using:
- Training: Vimeo-90K and BVC-DVC datasets.
- Testing/Evaluation: The AVS common test sequences (HUD4K, 1080p, 720p), the HEVC dataset (Classes B, C, D, and E), UVG, and MCL-JCV.
Key Insights and Ideas
- Skip Mode is Dominant: The authors show that in NVC, skip ratios are incredibly high—often over 90% for motion vectors at low bitrates. This makes optimizing the skip decision one of the highest-leverage ways to improve overall efficiency.
- Sensitivity Matters: Some feature channels are "information-rich" and highly sensitive to noise, while others are robust. A fixed threshold ignores this, either wasting bits on robust channels or introducing artifacts in sensitive ones.
- The Reference Quality Ripple Effect: In hierarchical coding structures, the distortion distribution changes significantly based on reference frame quality. RDOSkip is the first work to explicitly model this temporal dependency for NVC skip modes.
Key Results
- BD-Rate Savings: The method achieved an average of 0.90% BD-rate saving on AVS test sequences and 1.29% on HEVC sequences compared to the baseline AVS-EEM codec.
- Standardization: The temporal skip mode decision component was successful enough to be adopted into the ongoing AVS NVC standard (as contribution M8789).
- Complexity: The computational overhead is negligible. In fact, because the method skips more "unimportant" feature elements more accurately, it can slightly increase decoding speed (FPS) by reducing the workload for entropy decoding.
Limitations and Future Work
- Heuristic Hyperparameters: The adaptive thresholds rely on four hyperparameters ($\alpha, \beta, \gamma, \epsilon$) that were empirically set. While they proved robust across different GOP sizes and datasets, a more automated or learned way to determine these per-model could be beneficial.
- Focus on Residuals: This paper primarily focuses on skip mode for residual features. While motion vector skip ratios are already very high, applying a similar RD-optimized logic to motion features specifically might yield further (albeit smaller) gains.
Summary updated