TL;DR
This paper introduces InstA (Instance-Adaptive video compression), a method that finetunes a neural video codec on the specific video sequence it is compressing. By efficiently encoding and transmitting these specialized weight updates alongside the video data, the authors achieve up to 27% bitrate savings and enable decoders that are 70% smaller without sacrificing quality.
What Problem Does This Paper Address?
Traditional neural video codecs are "global" models: they are trained on a massive dataset and then frozen. At test time, they are expected to generalize to any video sequence. However, this approach faces three major hurdles:
- The Generalization Gap: A model trained on natural landscapes often performs poorly on animated content or high-motion sports due to "domain shift."
- Model Capacity: To handle every possible type of video, neural codecs must be large and computationally expensive, which is a nightmare for mobile deployment.
- Suboptimal Fitting: A global model is a "jack of all trades, master of none." It rarely finds the mathematically optimal parameters for a specific 10-second clip.
The authors propose that instead of building a better "universal" model, we should allow the model to "overfit" to the specific video it needs to compress.
Methods
The core of the InstA approach is a two-step process at the encoder:
- Instance-Adaptive Finetuning: Starting from a pretrained global model, the encoder performs a few iterations of gradient descent on the target video sequence to minimize a rate-distortion loss.
- Parameter Compression: The encoder doesn't send the entire set of new weights. Instead, it transmits the difference ($\delta$) between the global weights and the finetuned weights.
To make this transmission efficient, the authors use a spike-and-slab prior for entropy coding. This mixture model consists of a very narrow "spike" at zero and a wider "slab." This encourages the finetuning process to be sparse—only changing the weights that provide the most "bang for the buck" in terms of quality—while making zero-value updates essentially free to transmit.
This method is model-agnostic. The authors demonstrated it on two architectures: Scale-Space Flow (SSF) for P-frame (predictive) settings and B-EPIC for B-frame (bidirectional) settings.
Datasets
The global models were trained on the Vimeo90k dataset. Evaluation was performed across several standard benchmarks to test both general performance and robustness:
- UVG-1k: High-frame-rate (120fps) Full-HD sequences.
- HEVC Class B & C: Standard sequences used in traditional codec development.
- Xiph-5N: A collection of diverse 1080p sequences.
- Big Buck Bunny (BBB): Used specifically to test domain shift. The authors selected "easy" (BBB-E) and "hard" (BBB-H) segments from this animated film to see how a model trained on natural scenes handles animation.
Key Insights and Ideas
- Trading Compute for Rate: InstA introduces a "preset" capability to neural codecs. Just as you can tell a traditional encoder to take more time (e.g.,
veryslow) to get a better file size, InstA allows you to spend more GPU cycles finetuning to drive down the bitrate. - Specialization Enables Shrinkage: Because a finetuned model only needs to "know" one video, it doesn't need as much "brain power" (parameters). The authors found they could use a much smaller base model and still outperform a massive global model.
- Where Adaptation Matters: The authors observed that most of the "bit budget" for weight updates is spent on the I-frame (keyframe) model and the components closest to the final image output (the decoder), rather than the internal "hyper-prior" networks.
Key Results
The quantitative gains are substantial across the board:
- Bitrate Savings: InstA-SSF achieved 21% to 27% BD-rate savings over its own global baseline. Against the popular H.265 (x265) implementation, it showed savings between 5% and 44%.
- Model Reduction: The authors tested a "mini" version of their model (SSF5). Despite having 70% lower computational complexity (kMACs/pixel) than the full SSF18 model, the adaptive SSF5 outperformed the global SSF18.
- Robustness: On the animated Big Buck Bunny sequences, the global model struggled significantly. InstA was able to "learn" the animation style on the fly, closing the gap with classical codecs that usually handle domain shifts more gracefully.
Limitations and Future Work
- Encoder Complexity: The primary drawback is the encoding time. Finetuning on the test set is computationally expensive, making this currently unsuitable for live broadcasting or video calls. It is, however, a perfect fit for "one-to-many" scenarios like Netflix or YouTube streaming, where a video is compressed once and watched millions of times.
- Update Frequency: The paper primarily explores finetuning on short clips (10 seconds). Determining the optimal frequency for sending weight updates in a feature-length film remains an open question.
- Quantization: The authors used a relatively simple uniform quantization for the weight updates. Future work could explore more sophisticated model compression techniques to further reduce the overhead of sending new parameters.
Summary updated