Abstract
In this work we propose a novel deep learning approach for ultra-low bitrate video compression for video conferencing applications. To address the shortcomings of current video compression paradigms when the available bandwidth is extremely limited, we adopt a model-based approach that employs deep neural networks to encode motion information as keypoint displacement and reconstruct the video signal at the decoder side. The overall system is trained in an end-to-end fashion minimizing a reconstruction error on the encoder output. Objective and subjective quality evaluation experiments demonstrate that the proposed approach provides an average bitrate reduction for the same visual quality of more than 80% compared to HEVC.
Introduction
Standard video codecs like H.264 and HEVC rely on a "hybrid" paradigm involving block-based motion compensation and transform coding. While highly optimized, these methods struggle at ultra-low bitrates (below 10 Kbps), often resulting in severe blocking artifacts and loss of structural integrity. This paper addresses the specific constraints of video conferencing—typically featuring a static background and a single "talking head"—by shifting from pixel-level fidelity to a semantic, model-based compression strategy.
Methodology
The proposed system utilizes Deep Image Animation to model content rather than individual pixels. The architecture is divided into an encoder and a decoder pipeline.
Encoder Pipeline
- Intra-frame Coding: The initial frame ($F_0$) is compressed using BPG (Better Portable Graphics) to serve as a high-quality reference anchor.
- Keypoint Extraction: For subsequent frames, a U-Net architecture extracts $M$ sparse 2D keypoints. Each keypoint is accompanied by a $2 \times 2$ Jacobian matrix to capture local orientation and scale.
- Entropy Coding: These 5-dimensional vectors are compressed using a standard LZW encoder (
gzip).
Decoder Pipeline
The decoder receives the reference frame and keypoints, using a reconstruction network to estimate optical flow and warp the reference frame to match the current pose and expression. To prevent quality drift, an Adaptive Intra-Refresh scheme triggers a new reference frame if the reconstruction quality (PSNR) falls below a specific threshold.
Evaluation and Results
The model was trained on the VoxCeleb dataset and evaluated using "talking head" sequences from Xiph.org.
The results demonstrate a significant performance leap over traditional methods:
- Bitrate Efficiency: The method achieves an average BD-rate saving of over 80% compared to HEVC.
- Subjective Quality: Mean Opinion Score (MOS) tests indicate that users consistently prefer the proposed method over HEVC at bitrates between 5 and 55 Kbps.
- Visual Fidelity: Unlike HEVC, which becomes heavily distorted at extremely low bitrates (e.g., 2.3 Kbps), this approach maintains sharp facial features and natural movement.
Key Insights and Limitations
The core innovation lies in prioritizing semantic realism over mathematical pixel accuracy. By utilizing Jacobian matrices, the model efficiently represents complex deformations like eye squinting or mouth stretching.
However, challenges remain for deployment. The current implementation is limited to $256 \times 256$ resolution, and the computational cost of running deep generative networks at the decoder is significantly higher than hardware-accelerated HEVC. Future work will focus on scaling to higher resolutions, handling multiple speakers, and refining Rate-Distortion Optimization (RDO) for Intra-refresh selection.
Summary updated