Abstract
Existing codecs are designed to eliminate intrinsic redundancies to create a compact representation for compression. However, strong external priors from Multimodal Large Language Models (MLLMs) have not been explicitly explored in video compression. Herein, we introduce a unified paradigm for Cross-Modality Video Coding (CMVC), which is a pioneering approach to explore multimodality representation and video generative models in video coding. Specifically, on the encoder side, we disentangle a video into spatial content and motion components, which are subsequently transformed into distinct modalities to achieve very compact representation by leveraging MLLMs. During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes that optimize video reconstruction quality for specific decoding requirements, including Text-Text-to-Video (TT2V) mode to ensure high-quality semantic information and Image-Text-to-Video (IT2V) mode to achieve superb perceptual consistency. In addition, we propose an efficient frame interpolation model for IT2V mode via Low-Rank Adaption (LoRA) tuning to guarantee perceptual quality, which allows the generated motion cues to behave smoothly. Experiments on benchmarks indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency. These results highlight potential directions for future research in video coding.
Introduction
Traditional video compression standards like H.264, HEVC, and VVC rely on signal-level reconstruction, aiming to minimize pixel-wise differences. While effective at high bitrates, these methods suffer from severe artifacts—such as blocking and blurring—at ultra-low bitrates. In scenarios like remote surveillance or disaster response, preserving semantic meaning ("what is happening") is often more critical than exact pixel fidelity. The Cross-Modality Video Coding (CMVC) framework addresses this by shifting the paradigm from pixel compression to semantic compression using Multimodal Large Language Models (MLLMs).
The CMVC Framework
The CMVC framework treats video as a combination of spatial content and temporal dynamics. The process begins at the encoder with disentanglement.
Encoder: Disentanglement and Representation
The encoder dynamically selects keyframes using a CLIP-based encoder to identify significant scene changes. These components are then converted into compact multimodal representations:
- Spatial components are represented as images or descriptive text.
- Temporal components are summarized into motion-descriptive text using MLLMs like VideoLLaVA.
Decoder: Flexible Generative Reconstruction
The decoder offers two primary modes to balance bitrate and quality:
- TT2V (Text-Text-to-Video): Uses text for both spatial and temporal cues, achieving extreme compression while maintaining high semantic quality.
- IT2V (Image-Text-to-Video): Uses keyframe images and motion text to ensure better perceptual consistency and visual anchoring.
LoRA-based Frame Interpolation
A significant challenge in generative video coding is temporal "jumpiness." To ensure smooth transitions, CMVC employs a novel frame interpolation model based on Low-Rank Adaptation (LoRA). By fine-tuning a Stable Diffusion UNet on specific keyframes at the encoder, small LoRA weights are transmitted to the decoder. The decoder then performs model interpolation, blending the learned knowledge of adjacent frames to generate semantically consistent intermediate frames.
Experimental Results
CMVC was evaluated on standard benchmarks including HEVC Test Sequences, UVG, and MCL-JCV.
- Compression Efficiency: In IT2V mode, CMVC achieved substantial BD-rate reductions compared to the HEVC reference software (VTM), with improvements of -59.12% on Class B and -52.16% on Class D.
- Perceptual Quality: Qualitative analysis demonstrates that CMVC maintains sharp, semantically accurate visuals at bitrates where traditional codecs produce unwatchable results. The framework prioritizes metrics like DISTS, which better align with human perception than traditional PSNR.
Conclusion
CMVC represents a significant leap toward semantic-aware video coding. By leveraging the generative power of MLLMs and diffusion models, it achieves high-quality reconstruction at ultra-low bitrates. Future work will focus on reducing computational complexity and further improving temporal consistency for longer sequences.
Summary updated