EAD-Net: Emotion-Aware Talking Head Generation with Spatial Refinement and Temporal Coherence

Yahui Li, Yinfeng Yu, Liejun Wang, Shengjie Shen

TL;DR

EAD-Net is a novel diffusion-based framework for generating highly expressive, emotionally accurate talking head videos from audio. By combining fine-grained text-audio semantics with efficient strip-based attention and graph neural networks, it achieves precise lip-syncing and natural facial expressions while maintaining excellent temporal smoothness across long video sequences.

What Problem Does This Paper Address?

Creating realistic talking head videos driven by audio has come a long way, but adding realistic emotion remains a major hurdle. Existing methods typically suffer from three core issues:

  1. Coarse Emotional Control: Many models rely on simple, discrete emotion labels (e.g., "happy", "sad"), which fail to capture the subtle, continuous dynamics of human expression.
  2. The Lip-Sync Trade-Off: Introducing complex, high-level emotional semantics often degrades the accuracy of lip synchronization.
  3. Temporal Incoherence: Mainstream video generation models struggle to maintain global motion awareness over long sequences, frequently resulting in visual artifacts, motion jitter, and identity drift.

To bridge this gap, the authors introduce EAD-Net (Emotion-Aware Diffusion Network), a framework designed to balance fine-grained emotional expressiveness, strict audio-lip synchronization, and computational efficiency.

Methods

EAD-Net uses a denoising U-Net architecture as its backbone diffusion model, incorporating three novel components to handle spatial refinement and temporal coherence:

Overview of the EAD-Net framework
  • Spatio-Temporal Directional Attention (STDA): Standard self-attention scales quadratically with feature map size, making long video generation computationally prohibitive. STDA replaces full self-attention with orthogonal (vertical and horizontal) strip attention operators. This achieves a global receptive field with linear complexity, capturing fine-grained pixel correlations without exploding memory usage.
  • Temporal Frame Graph Reasoning Module (TFRM): Rather than relying on implicit attention weights to model time, TFRM explicitly structures the video frames as a bidirectional chain graph. A Graph Neural Network (GNN) processes these frame nodes to enforce continuous motion trajectories and smooth transitions, mitigating inter-frame jitter.
  • Text-Audio Fusion & Quality Filtering: Audio alone can be emotionally ambiguous. To resolve this, the authors use a large vision-language model (Qwen-VLM) to extract detailed textual descriptions of facial expressions from training videos. These text embeddings (processed via a T5 encoder) are fused with audio features (processed via Whisper) to provide clean, high-level semantic guidance. A strict cosine-similarity filter removes poorly aligned training samples to ensure data quality.

The model is optimized using a multi-task loss function combining standard noise prediction, an extended 16-frame SyncNet loss for lip synchronization, LPIPS for visual detail, and a TREPA (Temporal Representation Alignment) loss for sequence-level consistency.

Datasets

The model was trained and evaluated on the following datasets:

  • HDTF (High-Definition Talking Face): 362 high-definition videos ranging from 720p to 1080p, used to evaluate general talking head generation and lip-sync accuracy.
  • MEAD: A large-scale emotional talking face dataset featuring 60 actors speaking with 8 different emotions at 3 levels of intensity. This was used to evaluate emotional accuracy.
  • RAVDESS: A dynamic, multimodal dataset of emotional speech and song, utilized strictly for cross-dataset generalization testing to prove the model's robustness on unseen identities.

Key Insights and Ideas

  • Text as an Emotional Anchor: Audio rhythm and pitch carry emotional cues, but text descriptions provide explicit semantic boundaries. Fusing text priors with audio features prevents the model from drifting into ambiguous expressions.
  • Decoupling High and Low Frequencies: The ablation studies revealed an interesting representational clash: TFRM suppresses high-frequency inter-frame variations to ensure smoothness, while STDA relies on those variations for fine-grained pixel alignment. Introducing global semantic text layouts bypassed this bottleneck, allowing STDA to integrate semantic priors with mid-to-low-frequency features smoothly.
  • Explicit vs. Implicit Time Modeling: Relying solely on temporal attention often fails in long-form video generation. Explicitly treating frames as nodes in a structured graph forces the network to respect physical continuity across time.

Key Results

EAD-Net outpaced state-of-the-art (SOTA) methods across both quantitative metrics and qualitative evaluations:

Quantitative Performance

  • Lip-Sync Accuracy: On the HDTF dataset, EAD-Net achieved a SOTA SyncNet confidence score of 9.04, outperforming competitive diffusion models like LatentSync (8.91) and Hallo2 (5.82).
  • Temporal Consistency: EAD-Net achieved the lowest Fréchet Video Distance (FVD of 312.347 on HDTF and 265.082 on MEAD), proving its superior temporal smoothness.
  • Emotional Accuracy: On the MEAD benchmark, the model reached an emotion classification accuracy ($\text{Acc}_{\text{emo}}$) of 46.67%, validating the effectiveness of the VLM-guided text-audio fusion strategy.

Qualitative Advantages

Visually, EAD-Net successfully avoids the mouth-area artifacts common in GAN-based approaches like Wav2Lip. It also preserves identity features and gaze directions much more reliably than recent diffusion alternatives, which frequently suffer from blurry teeth or unnatural eye distortions when rendering intense emotions like anger.

Qualitative comparison against SOTA methods

Limitations and Future Work

Despite its strong performance, the authors note a few limitations:

  • Discrete Emotion Constraints: The model currently only supports eight predefined emotion categories with fixed intensities, lacking truly continuous control over emotional variance.
  • Frontal View Bias: The current implementation focuses primarily on frontal viewpoints and lacks robust head pose control.

Future Directions: The team plans to explore continuous emotion modeling using valence and arousal dimensions to represent arbitrary emotional states. Additionally, they aim to integrate end-to-end head pose generation using 3D representations like Gaussian Splatting to enhance pose diversity and realism.

Summary updated

Abstract

Emotionally talking head video generation aims to generate expressive portrait videos with accurate lip synchronization and emotional facial expressions. Current methods rely on simple emotional labels, leading to insufficient semantic information. While introducing high-level semantics enhances expressiveness, it easily causes lip-sync degradation. Furthermore, mainstream generation methods struggle to balance computational efficiency and global motion awareness in long videos and suffer from poor temporal coherence. Therefore, we propose an \textbf{E}motion-\textbf{A}ware \textbf{D}iffusion model-based \textbf{Net}work, called \textbf{EAD-Net}. We introduce SyncNet supervision and Temporal Representation Alignment (TREPA) to mitigate lip-sync degradation caused by multi-modal fusion. To model complex spatio-temporal dependencies in long video sequences, we propose a Spatio-Temporal Directional Attention (STDA) mechanism that captures global motion patterns through strip attention. Additionally, we design a Temporal Frame graph Reasoning Module (TFRM) to explicitly model temporal coherence between video frames through graph structure learning. To enhance emotional semantic control, a large language model is employed to extract textual descriptions from real videos, serving as high-level semantic guidance. Experiments on the HDTF and MEAD datasets demonstrate that our method outperforms existing methods in terms of lip-sync accuracy, temporal consistency, and emotional accuracy.
BibTeX
@inproceedings{li2026ead,
  author = {Yahui Li and Yinfeng Yu and Liejun Wang and Shengjie Shen},
  title = {{EAD-Net: Emotion-Aware Talking Head Generation with Spatial Refinement and Temporal Coherence}},
  booktitle = {ICMR},
  year = {2026},
  doi = {10.1145/3805622.3810860},
}

Loading related content...

Figure