LawDNet: Enhanced Audio-Driven Lip Synthesis via Local Affine Warping Deformation

Junli Deng, Yihao Luo, Xueting Yang, Li Siyou, Wei Wang, Jinyang Guo, Ping Shi

Abstract

In the domain of photorealistic avatar generation, the fidelity of audio-driven lip motion synthesis is essential for realistic virtual interactions. Existing methods face two key challenges: a lack of vivacity due to limited diversity in generated lip poses and noticeable anamorphose motions caused by poor temporal coherence. To address these issues, we propose LawDNet, a novel deep-learning architecture enhancing lip synthesis through a Local Affine Warping Deformation mechanism. This mechanism models the intricate lip movements in response to the audio input by controllable non-linear warping fields. These fields consist of local affine transformations focused on abstract keypoints within deep feature maps, offering a novel universal paradigm for feature warping in networks. Additionally, LawDNet incorporates a dual-stream discriminator for improved frame-to-frame continuity and employs face normalization techniques to handle pose and scene variations. Extensive evaluations demonstrate LawDNet's superior robustness and lip movement dynamism performance compared to previous methods. The advancements presented in this paper, including the methodologies, training data, source codes, and pre-trained models, will be made accessible to the research community.

Introduction: The Challenge of Realistic Lip Synthesis

Generating photorealistic talking heads from audio requires balancing expressive movement with structural stability. Current approaches typically fall into two categories: direct generation and warping-based methods. Direct generation models often struggle with temporal coherence, leading to jittery artifacts. Warping-based methods, while more stable, either suffer from unnatural distortions (dense flow) or lack the necessary detail for expressive speech (global affine).

LawDNet synthesizes lip-shape regenerated video by combining a source video with driven audio.

LawDNet introduces a "Goldilocks" solution: the Local Affine Warping Deformation (LawD) mechanism. This approach is flexible enough to capture intricate muscle movements while remaining stable and computationally efficient.

Methodology: Local Affine Warping Deformation

The core innovation of LawDNet is the LawD mechanism. Instead of predicting movement for every pixel, the model identifies abstract, self-learned keypoints in the lip area that correspond to audio-related muscle movements. Each keypoint is assigned a local affine transformation (rotation, scaling, translation) and an "impact radius," allowing the model to "pull" and "shape" specific parts of the mouth independently.

The LawD architecture uses keypoint and radius detectors to generate a coarse-grained grid deformation for warping feature maps.

This "Learnable Liquify" concept uses matrix exponentials to define non-linear warping fields, modeling complex mouth shapes like pursing or stretching more effectively than linear models. To ensure high-quality output, LawDNet employs a dual-stream discriminator: a spatial discriminator for per-frame realism and a temporal discriminator to ensure smooth motion across frames. Additionally, a face frontalization preprocessing step aligns faces to a standard view, simplifying the learning task and improving robustness against extreme head poses.

Experiments and Results

The model was evaluated on the HDTF benchmark and the new Chinese Authentic Talking Head Dataset (CATHD), which features over 40 hours of "in-the-wild" footage from 400+ subjects. LawDNet outperformed state-of-the-art models like Wav2Lip and DINet across multiple metrics.

On the HDTF dataset, it achieved a 3.07% increase in SSIM (structural similarity) and a 6.33% improvement in LSE-D (Lip Sync Error - Distance), indicating superior synchronization and structural fidelity. Qualitative results highlight sharper lip boundaries and more vivid movements. Ablation studies confirmed that the temporal discriminator was vital; without it, the lips suffered from distracting jitter.

Conclusion and Future Work

LawDNet represents a significant step forward in audio-driven lip synthesis by combining local flexibility with global stability. By focusing on "audio-related muscles" through self-learned keypoints rather than fixed landmarks, the model achieves high-quality results with fewer parameters. While occasional edge distortions in masked areas remain a challenge, the model's efficiency and robustness make it a powerful tool for virtual interactions. Future work will explore integrating audio-to-3D model conversion and applying these warping techniques to full facial reenactment.

Summary updated

Abstract

In the domain of photorealistic avatar generation, the fidelity of audio-driven lip motion synthesis is essential for realistic virtual interactions. Existing methods face two key challenges: a lack of vivacity due to limited diversity in generated lip poses and noticeable anamorphose motions caused by poor temporal coherence. To address these issues, we propose LawDNet, a novel deep-learning architecture enhancing lip synthesis through a Local Affine Warping Deformation mechanism. This mechanism models the intricate lip movements in response to the audio input by controllable non-linear warping fields. These fields consist of local affine transformations focused on abstract keypoints within deep feature maps, offering a novel universal paradigm for feature warping in networks. Additionally, LawDNet incorporates a dual-stream discriminator for improved frame-to-frame continuity and employs face normalization techniques to handle pose and scene variations. Extensive evaluations demonstrate LawDNet's superior robustness and lip movement dynamism performance compared to previous methods. The advancements presented in this paper, including the methodologies, training data, source codes, and pre-trained models, will be made accessible to the research community.
BibTeX
@inproceedings{deng2025lawdnet,
  author = {Junli Deng and Yihao Luo and Xueting Yang and Li Siyou and Wei Wang and Jinyang Guo and Ping Shi},
  title = {{LawDNet: Enhanced Audio-Driven Lip Synthesis via Local Affine Warping Deformation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=r0g1bvwbFR},
}
Figure