Ultra-Low Bitrate Video Conferencing Using Deep Image Animation

Goluck Konuko, Giuseppe Valenzise, Stéphane Lathuilière

TL;DR

This paper introduces a novel video conferencing codec that achieves over 80% bitrate reduction compared to HEVC by shifting from pixel-based compression to a generative, model-based approach. Instead of encoding pixel residuals, the system transmits sparse facial keypoints and uses a deep generative network at the decoder to "animate" a high-quality reference frame.

What Problem Does This Paper Address?

Standard video codecs like H.264 and HEVC (and even the newer VVC) are built on a "hybrid" paradigm that has dominated for 30 years: they use block-based motion compensation and transform coding (like DCT) to minimize pixel-level errors. While highly optimized, these methods break down at ultra-low bitrates (e.g., below 10 Kbps). When bandwidth is extremely limited, traditional codecs produce "blocky" artifacts and lose structural integrity because they prioritize pixel-fidelity over semantic realism.

The authors target video conferencing specifically because it usually features a static background and a single "talking head." This predictability allows for a more aggressive compression strategy: modeling the content rather than the pixels.

Methods

The authors propose a pipeline based on Deep Image Animation. The system is split into an encoder and a decoder:

  1. Intra-frame Coding: The first frame ($F_0$) is compressed using BPG (Better Portable Graphics), which is essentially the Intra-coding mode of HEVC. This provides a high-quality reference "anchor."
  2. Keypoint Extraction: For subsequent frames, a U-Net architecture predicts a set of $M$ sparse 2D keypoints. Crucially, each keypoint includes a $2 \times 2$ Jacobian matrix that captures the local orientation and scale of the facial region.
  3. Entropy Coding: These keypoints (represented as 5-dimensional vectors) are compressed using a standard LZW encoder (gzip).
  4. Motion Compensation & Reconstruction: The decoder receives the reference frame and the keypoints. It uses a reconstruction network (consisting of two sub-networks) to estimate optical flow and warp the reference frame to match the current expression and pose.
  5. Adaptive Intra-Refresh: To prevent quality drift over long sequences (e.g., when a person turns their head significantly), the authors implemented an "Adaptive Intra frame selection" scheme. The encoder maintains a buffer of frames; if the reconstruction quality (PSNR) drops below a threshold $\tau$, it triggers the transmission of a new Intra-coded reference frame.

Datasets

The model was trained and evaluated using two primary sources:

  • VoxCeleb: A massive dataset of over 22,000 YouTube videos of people speaking. The authors filtered and resized these to $256 \times 256$ resolution.
  • Xiph.org: A classic repository for video processing research. The authors selected 16 "talking head" sequences (like the well-known "Akiyo" or "Salesman" clips) to test the model's generalization on data it hadn't seen during training.

Key Insights and Ideas

  • Semantics over Pixels: The core "aha" moment is that at ultra-low bitrates, human viewers care more about a face looking "real" and moving naturally than they do about exact pixel-to-pixel mathematical accuracy (MSE).
  • The Power of the Jacobian: By including the Jacobian matrix with each keypoint, the model can represent complex local deformations (like the squinting of an eye or the stretching of a mouth) with very few parameters.
  • Open-Loop Efficiency: Unlike traditional codecs that operate in a "closed-loop" (constantly checking the error between the original and the reconstructed frame to adjust), this system is largely open-loop, relying on the generative power of the decoder to fill in the blanks.
  • Hybridizing Old and New: The authors didn't reinvent everything; they used BPG/HEVC for the heavy lifting of the initial frame and gzip for the keypoints, focusing their innovation on the neural motion modeling.

Key Results

The results are striking, particularly in the "starvation" regime of low bandwidth:

  • Bitrate Savings: The proposed method achieved an average BD-rate saving of over 80% compared to HEVC. This means it can deliver the same visual quality as HEVC while using only one-fifth of the data.
  • Subjective Quality: In Mean Opinion Score (MOS) tests conducted via Amazon Mechanical Turk, users consistently rated the proposed method higher than HEVC at bitrates between 5 and 55 Kbps.
  • Visual Integrity: Qualitatively, while HEVC becomes a blurry mess of blocks at 2.3 Kbps, the proposed method maintains sharp features (like eyes and teeth), even if the reconstruction isn't a perfect pixel-match to the source.

Limitations and Future Work

Despite the impressive gains, the authors acknowledge several hurdles:

  • Resolution: The current implementation is limited to $256 \times 256$ pixels. Scaling this to HD or 4K while maintaining real-time performance is a significant challenge.
  • Complexity: The model currently focuses on a single speaker. Handling multiple speakers or complex backgrounds would require more sophisticated ROI (Region of Interest) management.
  • Rate-Distortion Optimization: The current system uses a fixed threshold for Intra-refresh. The authors suggest that a more formal Rate-Distortion Optimization (RDO) process—deciding mathematically when it is "worth it" to send a new keyframe—could further improve efficiency.
  • Computational Cost: While the bitrate is low, the computational power required to run deep generative networks at the decoder is much higher than traditional hardware-accelerated HEVC decoding.

Summary updated

Abstract

In this work we propose a novel deep learning approach for ultra-low bitrate video compression for video conferencing applications. To address the shortcomings of current video compression paradigms when the available bandwidth is extremely limited, we adopt a model-based approach that employs deep neural networks to encode motion information as keypoint displacement and reconstruct the video signal at the decoder side. The overall system is trained in an end-to-end fashion minimizing a reconstruction error on the encoder output. Objective and subjective quality evaluation experiments demonstrate that the proposed approach provides an average bitrate reduction for the same visual quality of more than 60% compared to HEVC.
BibTeX
@inproceedings{konuko2022ultra,
  author = {Goluck Konuko and Giuseppe Valenzise and Stéphane Lathuilière},
  title = {{Ultra-Low Bitrate Video Conferencing Using Deep Image Animation}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=yF35GKKzmA},
}

Loading related content...

Figure