Abstract
Deep generative models, and particularly facial animation schemes, can be used in video conferencing applications to efficiently compress a video through a sparse set of keypoints, without the need to transmit dense motion vectors. While these schemes bring significant coding gains over conventional video codecs at low bitrates, their performance saturates quickly when the available bandwidth increases. In this paper, we propose a layered, hybrid coding scheme to overcome this limitation. Specifically, we extend a codec based on facial animation by adding an auxiliary stream consisting of a very low bitrate version of the video, obtained through a conventional video codec (e.g., HEVC). The animated and auxiliary videos are combined through a novel fusion module. Our results show consistent average BD-Rate gains in excess of -30% on a large dataset of video conferencing sequences, extending the operational range of bitrates of a facial animation codec alone.
Introduction
In the current landscape of digital communication, video conferencing has become an essential tool for both professional and personal interaction. However, maintaining high visual quality under constrained bandwidth remains a significant challenge. Traditional video coding standards, such as HEVC (H.265) and the more recent VVC (H.266), rely on block-based motion compensation and residual coding. While effective at moderate bitrates, these methods often produce severe blocking artifacts and blurring when pushed to ultra-low bitrates (e.g., below 20 kbps).
To address this, researchers have turned to deep generative models, specifically facial animation codecs. These models transmit a single reference frame followed by a sparse set of keypoints that describe facial movements, allowing the receiver to "animate" the source image. While these methods achieve impressive sharpness at extremely low bitrates, they suffer from a "quality ceiling"—their performance plateaus even as more bandwidth is provided. Furthermore, they often struggle with temporal drift, background stability, and the presence of non-facial objects like hands or headsets. This paper introduces H-DAC (Hybrid Deep Animation Codec), a system designed to bridge the gap between generative animation and traditional pixel-based coding.
Methodology
The H-DAC architecture is a layered, hybrid system that integrates three primary components to ensure both efficiency and robustness.
1. The Animation Module
The foundation of the system is the Deep Animation Codec (DAC). In this module, an initial Intra frame is compressed using standard methods (BPG or HEVC). For subsequent frames, the system extracts and transmits only 10 sparse facial keypoints. These coordinates are used by a generator network to warp the initial source frame, effectively synthesizing the speaker's current expression and head pose.
2. The Auxiliary Stream
The core innovation of H-DAC is the inclusion of an auxiliary stream. Simultaneously with the animation data, the system transmits a standard HEVC video stream compressed at a very high quantization level. Although this auxiliary stream is visually degraded on its own, it provides a critical "structural hint" of the entire frame, including the background and any objects not covered by the facial keypoint model.
3. The Fusion Module
The final component is a novel neural fusion module utilizing an encoder-decoder architecture with spatial attention layers. This module receives two inputs: the warped frame from the animation module and the low-quality frame from the HEVC auxiliary stream. The fusion network is trained to extract high-frequency facial details from the animation while using the HEVC stream to regularize the geometry, correct motion errors, and reconstruct non-facial elements. The system is optimized end-to-end using a combination of reconstruction loss, perceptual losses (msVGG and LPIPS), and adversarial losses to ensure realistic output.
Experimental Setup
The performance of H-DAC was evaluated using two significant datasets:
- VoxCeleb2: A large-scale dataset containing over 22,000 "talking head" videos. The researchers selected 90 sequences characterized by complex motion for rigorous testing.
- Xiph.org: A standard industry benchmark. Sixteen sequences featuring talking humans were used to test the model's ability to generalize across different environments and lighting conditions.
Results and Discussion
The hybrid approach yielded significant improvements over both pure animation models and traditional codecs. H-DAC achieved average BD-Rate gains exceeding -30% compared to standard HEVC, meaning it provides the same visual quality at a 30% lower bitrate.
Overcoming the Quality Ceiling
One of the most important findings is H-DAC's ability to scale. Unlike pure animation codecs that saturate quickly, H-DAC’s quality continues to improve as more bandwidth is allocated to the auxiliary stream. This allows the codec to remain competitive across a wider range of bitrates.
Robustness and Fidelity
Qualitative analysis demonstrated that H-DAC is significantly more robust than its predecessors. By utilizing the HEVC "hint," the fusion module can successfully reconstruct objects that pure facial models typically ignore or erase, such as hands moving across the face or microphones. The auxiliary stream acts as a structural anchor, preventing the generative model from "drifting" into unrealistic facial reconstructions over time.
Compared to VVC, the current industry gold standard, H-DAC performed similarly in objective metrics like PSNR but showed superior qualitative results at 20 kbps, producing sharper facial features and fewer distracting blocking artifacts.
Conclusion
H-DAC represents a significant step forward in low-bitrate video communication by successfully fusing generative animation with traditional video coding. While the system requires more computational power than standard codecs due to its reliance on deep neural networks, the trade-off in bitrate efficiency and visual clarity is substantial. Future work will explore the integration of VVC as the auxiliary stream and conduct extensive human subjective testing to further validate the perceived quality of these hybrid synthesized videos.
Summary updated