Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set

Ties van Rozendaal, Johann Brehmer, Yunfan Zhang, Reza Pourreza, Auke Wiggers, Taco S. Cohen

TL;DR

This paper introduces InstA (Instance-Adaptive video compression), a method that finetunes a neural video codec on the specific video sequence it is compressing. By efficiently encoding and transmitting these specialized weight updates alongside the video data, the authors achieve up to 27% bitrate savings and enable decoders that are 70% smaller without sacrificing quality.

What Problem Does This Paper Address?

Traditional neural video codecs are "global" models: they are trained on a massive dataset and then frozen. At test time, they are expected to generalize to any video sequence. However, this approach faces three major hurdles:

  1. The Generalization Gap: A model trained on natural landscapes often performs poorly on animated content or high-motion sports due to "domain shift."
  2. Model Capacity: To handle every possible type of video, neural codecs must be large and computationally expensive, which is a nightmare for mobile deployment.
  3. Suboptimal Fitting: A global model is a "jack of all trades, master of none." It rarely finds the mathematically optimal parameters for a specific 10-second clip.

The authors propose that instead of building a better "universal" model, we should allow the model to "overfit" to the specific video it needs to compress.

Methods

The core of the InstA approach is a two-step process at the encoder:

  1. Instance-Adaptive Finetuning: Starting from a pretrained global model, the encoder performs a few iterations of gradient descent on the target video sequence to minimize a rate-distortion loss.
  2. Parameter Compression: The encoder doesn't send the entire set of new weights. Instead, it transmits the difference ($\delta$) between the global weights and the finetuned weights.

To make this transmission efficient, the authors use a spike-and-slab prior for entropy coding. This mixture model consists of a very narrow "spike" at zero and a wider "slab." This encourages the finetuning process to be sparse—only changing the weights that provide the most "bang for the buck" in terms of quality—while making zero-value updates essentially free to transmit.

This method is model-agnostic. The authors demonstrated it on two architectures: Scale-Space Flow (SSF) for P-frame (predictive) settings and B-EPIC for B-frame (bidirectional) settings.

An overview of how InstA compares to other adaptation methods in the literature.

Datasets

The global models were trained on the Vimeo90k dataset. Evaluation was performed across several standard benchmarks to test both general performance and robustness:

  • UVG-1k: High-frame-rate (120fps) Full-HD sequences.
  • HEVC Class B & C: Standard sequences used in traditional codec development.
  • Xiph-5N: A collection of diverse 1080p sequences.
  • Big Buck Bunny (BBB): Used specifically to test domain shift. The authors selected "easy" (BBB-E) and "hard" (BBB-H) segments from this animated film to see how a model trained on natural scenes handles animation.

Key Insights and Ideas

  • Trading Compute for Rate: InstA introduces a "preset" capability to neural codecs. Just as you can tell a traditional encoder to take more time (e.g., veryslow) to get a better file size, InstA allows you to spend more GPU cycles finetuning to drive down the bitrate.
  • Specialization Enables Shrinkage: Because a finetuned model only needs to "know" one video, it doesn't need as much "brain power" (parameters). The authors found they could use a much smaller base model and still outperform a massive global model.
  • Where Adaptation Matters: The authors observed that most of the "bit budget" for weight updates is spent on the I-frame (keyframe) model and the components closest to the final image output (the decoder), rather than the internal "hyper-prior" networks.

Key Results

The quantitative gains are substantial across the board:

  • Bitrate Savings: InstA-SSF achieved 21% to 27% BD-rate savings over its own global baseline. Against the popular H.265 (x265) implementation, it showed savings between 5% and 44%.
  • Model Reduction: The authors tested a "mini" version of their model (SSF5). Despite having 70% lower computational complexity (kMACs/pixel) than the full SSF18 model, the adaptive SSF5 outperformed the global SSF18.
  • Robustness: On the animated Big Buck Bunny sequences, the global model struggled significantly. InstA was able to "learn" the animation style on the fly, closing the gap with classical codecs that usually handle domain shifts more gracefully.
Table 2: Parameter counts and decoding complexity for our various scale-space-flow base models. Compres- sion curves comparing all four models can be seen in the left panel of Fig. 7.

Limitations and Future Work

  • Encoder Complexity: The primary drawback is the encoding time. Finetuning on the test set is computationally expensive, making this currently unsuitable for live broadcasting or video calls. It is, however, a perfect fit for "one-to-many" scenarios like Netflix or YouTube streaming, where a video is compressed once and watched millions of times.
  • Update Frequency: The paper primarily explores finetuning on short clips (10 seconds). Determining the optimal frequency for sending weight updates in a feature-length film remains an open question.
  • Quantization: The authors used a relatively simple uniform quantization for the weight updates. Future work could explore more sophisticated model compression techniques to further reduce the overhead of sending new parameters.

Summary updated

Abstract

We introduce a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, we finetune a pretrained compression model. The optimal parameters are transmitted to the receiver along with the latent code. By entropy-coding the parameter updates under a suitable mixture model prior, we ensure that the network parameters can be encoded efficiently. This instance-adaptive compression algorithm is agnostic about the choice of base model and has the potential to improve any neural video codec. On UVG, HEVC, and Xiph datasets, our codec improves the performance of a scale-space flow model by between 21% and 27% BD-rate savings, and that of a state-of-the-art B-frame model by 17 to 20% BD-rate savings. We also demonstrate that instance-adaptive finetuning improves the robustness to domain shift. Finally, our approach reduces the capacity requirements of compression models. We show that it enables a competitive performance even after reducing the network size by 70%.
BibTeX
@article{rozendaal2023instance,
  author = {Ties van Rozendaal and Johann Brehmer and Yunfan Zhang and Reza Pourreza and Auke Wiggers and Taco S. Cohen},
  title = {{Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set}},
  journal = {Accepted by TMLR},
  year = {2023},
  url = {https://openreview.net/forum?id=akg6kdx0Pk},
}
Figure