Abstract
Traditional video compression focuses on pixel-level redundancy. This paper introduces Modality Transferable Visual Information Representation (MTVIR), a framework that transfers visual information into the parameters of a deep learning model. By serving as an enhancement layer for standard codecs like VVC, MTVIR achieves significant bitrate savings through joint rate-utility optimization of the model stream.
Introduction
Conventional video coding standards (H.264, HEVC, VVC) primarily target spatial and temporal redundancies within pixel data. While deep learning has enhanced specific modules like loop filtering, these approaches often treat neural networks as static tools. MTVIR proposes a paradigm shift: "modality transfer," where visual information is absorbed into the weights of a neural network trained online. The primary challenge lies in balancing the bitrate cost of transmitting model parameters against the quality gains they provide at the decoder.
The MTVIR Framework
The MTVIR framework partitions the video bitstream into a Base Layer (BL) and an Enhancement Layer (EL).
- Base Layer (BL): Utilizes standard Versatile Video Coding (VVC) to generate a baseline video stream.
- Enhancement Layer (EL): Transmits a compressed "Model Stream." This stream contains the weights of a Convolutional Neural Network (CNN) specifically trained to rectify distortions present in the Base Layer's reconstructed frames.
Key Innovations
Online Learning with Conv Quantization Blocks (CQB)
To ensure the model is inherently compressible, the authors utilize Conv Quantization Blocks (CQB). These blocks integrate scale transforms and quantization directly into the training loop using a straight-through estimator. This allows the model to learn weight distributions that are optimized for entropy coding without sacrificing reconstruction performance.
Inter-Model Redundancy Removal
Efficiency is further improved by transmitting only the residue—the difference between the online-trained model's weights and a universally initialized reference model—rather than the full weight set.
Rate-Utility Optimization
The core of MTVIR is its "Rate-Utility" optimization engine. The encoder evaluates the Lagrangian cost ($J = Distortion + \lambda \cdot Rate$) throughout the training process. It dynamically selects the model iteration that provides the optimal trade-off between the bits required for the model stream and the resulting PSNR improvement.
Experimental Results
The system was pre-trained on the DIV2K dataset and evaluated using VVC Common Test Conditions (CTC) sequences (Classes A-F). MTVIR was implemented atop VTM-4.0.
- Bitrate Efficiency: On average, the method achieved 2.63% BD-rate savings in All Intra (AI) mode and 4.07% in Random Access (RA) mode compared to standard VVC.
- Peak Performance: Savings reached up to 10.45% for specific sequences like "BQTerrace."
- Visual Quality: Qualitative analysis indicates a significant reduction in ringing and blocking artifacts, particularly in complex textures, compared to baseline VVC at equivalent bitrates.
Discussion and Conclusion
MTVIR demonstrates that neural networks can serve as compact representations of scene-specific knowledge. By training for the bitstream—incorporating quantization into the loss function—the model "anticipates" compression.
However, challenges remain. The computational overhead of online learning at the encoder is significant, and the relative overhead of the model stream is higher for low-resolution sequences. Future work will focus on more efficient architectures and faster adaptation techniques to bring modality transfer closer to real-time practical application.
Summary updated