Ultra-Low Bitrate Predictive Portrait Video Compression with Diffusion Models

Xinyi Chen, Weimin Lei, Wei Zhang, Yanwen Wang, Mingxin LiuSymmetry 2025 · DOI

TL;DR

This paper introduces a novel video compression framework that leverages diffusion models for ultra-low bitrate portrait video compression. By using a temporal diffusion predictor to generate most frames and an adaptive coding strategy to manage keyframes, the method achieves significant bitrate savings (over 24% compared to VVC) while maintaining high perceptual quality.

What Problem Does This Paper Address?

Traditional video compression standards like H.264/AVC, H.265/HEVC, and H.266/VVC have made significant strides in efficiency but still face challenges, especially at ultra-low bitrates. These standards often rely on manually designed components, limiting end-to-end optimization and struggling to capture complex, non-rigid motion characteristic of portrait videos. While deep learning has been integrated, achieving high-quality compression for portrait videos at extremely low bitrates remains an open problem. This paper aims to address this by developing a diffusion model-based approach that can effectively predict and reconstruct frames, minimizing the need for high-bitrate keyframes.

Methods

The core of the proposed method is a predictive portrait video compression framework that relies heavily on a temporal diffusion prediction model. This model is built upon a conditional latent diffusion model, which operates in a lower-dimensional latent space to reduce computational cost.

Key methodological components include:

  • Temporal Diffusion Prediction Model: This model uses a pre-trained Variational Autoencoder (VAE) to map image data to a latent representation. It then employs a diffusion process, specifically incorporating 3D causal convolution for short-term dynamics and a temporal attention mechanism for long-term dependencies, to predict future frames based on past keyframes.
  • Adaptive Coding Strategy: This strategy is crucial for managing the trade-off between compression efficiency and reconstruction quality. It consists of two parts:
  • Frame Quality Assessment: This module uses the Learned Perceptual Image Patch Similarity (LPIPS) metric to evaluate the perceptual quality of predicted frames. If a predicted frame falls below a predefined quality threshold, it and subsequent predicted frames are discarded, and only the high-quality predicted frames are retained.
  • Adaptive Keyframe Control: This component dynamically adjusts the length of the keyframe sequence based on the motion complexity of the video. Higher motion intensity leads to longer keyframe sequences, providing more context for prediction. This helps minimize keyframe transmission costs while maximizing reconstruction quality.

The framework is designed to be symmetric at both encoding and decoding ends, with the encoder deciding how many frames to predict and the decoder generating the same set of predicted frames.

Datasets

The authors used three portrait video datasets for training and evaluation:

  • Penn Action: Contains RGB video sequences of human actions.
  • TaiChiHD: Features TaiChi performance videos.
  • Fashion: Includes videos of a single model in diverse clothing.

All datasets were cropped and resized to 128x128 for experiments.

Key Insights and Ideas

  • Diffusion Models for Prediction: The paper demonstrates that diffusion models, typically used for generation, can be effectively adapted for predictive video compression, offering a powerful way to generate high-quality frames.
  • Temporal Dynamics are Key: Explicitly modeling temporal correlations using 3D causal convolutions and temporal attention is crucial for accurate video prediction, especially for portrait videos with complex motion.
  • Adaptive Keyframe Management: Dynamically adjusting keyframe length based on motion complexity is a critical innovation. This allows the system to be more efficient in low-motion scenes while retaining necessary information in high-motion scenes, striking a better balance between bitrate and quality.
  • Perceptual Quality Focus: The use of LPIPS and DISTS as evaluation metrics highlights a shift towards perceptual quality, which is often more relevant to human viewers than pixel-level metrics like PSNR, especially at low bitrates.

Key Results

  • Significant Bitrate Savings: The proposed method achieved over 24% bitrate savings compared to VVC in terms of perceptual distortion (DISTS) on the TaiChiHD dataset.
  • Superior Perceptual Quality: The method outperformed other predictive coding schemes (RDAC, Extreme) on perceptual metrics (DISTS, LPIPS) across various bitrates.
  • Competitive PSNR: While not always achieving the absolute highest PSNR, the method's PSNR performance was comparable to VVC at similar bitrates.
  • Efficient Encoding/Decoding: The proposed method demonstrated lower encoding and decoding times compared to VVC, partly due to DDIM acceleration.

Limitations and Future Work

The authors acknowledge several limitations:

  • Computational Complexity: The model's computational complexity may still be high, potentially hindering real-time applications.
  • Handling Complex Scenes: Further optimization is needed to handle highly complex scene variations.
  • Generalization: The model's generalization ability to diverse video content types could be improved.

Future work suggested includes:

  • Enhancing the model's generalization for broader applications.
  • Integrating the method with other video processing techniques like super-resolution and restoration.
  • Reducing computational costs for better real-time performance.

Summary updated

BibTeX
@article{chen2025ultra,
  author = {Xinyi Chen and Weimin Lei and Wei Zhang and Yanwen Wang and Mingxin Liu},
  title = {{Ultra-Low Bitrate Predictive Portrait Video Compression with Diffusion Models}},
  journal = {Symmetry},
  year = {2025},
  doi = {10.3390/sym17060913},
}
Figure