Hierarchical Semantic Perceptual Listener Head Video Generation: A High-performance Pipeline

Zhigang Chang, Weitai Hu, Qing Yang, Shibao Zheng

Abstract

Generating realistic listener head videos—where a digital character reacts naturally to a speaker—remains a significant challenge in computer vision due to the non-deterministic nature of human reactions. This paper presents a high-performance pipeline that combines hierarchical audio encoding, 3D-aware rendering, and video restoration to synthesize responsive listener behavior. By capturing multi-level semantic cues and ensuring temporal stability, the proposed method secured first place in the ViCo@2023 Conversational Head Generation Challenge, demonstrating superior performance in both motion accuracy and visual fidelity.

Introduction

While "Talking Head" generation has achieved remarkable realism by synchronizing lip movements with speech, the task of "Listening Head" generation is inherently more complex. In a conversation, a speaker's mouth movements are strictly governed by phonemes; however, a listener’s reactions—such as subtle nods, smiles, or shifts in gaze—are "ambiguous." There is no single correct response to a given utterance, but rather a range of socially appropriate non-verbal cues.

Effective listener generation requires a model to interpret not just the words spoken, but the tone, rhythm, and emotional subtext of the speaker. Furthermore, the resulting video must maintain high visual quality, avoiding the common pitfalls of distorted backgrounds or loss of identity during rapid head movements. This paper details a multi-stage pipeline designed to bridge the gap between speaker audio and realistic listener response.

Methodology

The proposed pipeline is structured into four distinct stages: feature extraction, motion prediction, rendering, and post-processing.

1. Hierarchical Semantic Encoding

To capture the nuances of human speech, the system employs a Hierarchical Audio Encoder inspired by the HA2G framework. Rather than processing audio as a monolithic data stream, the encoder extracts features at three levels:

  • Low-level: Capturing pitch and rhythm.
  • Mid-level: Identifying phonetic patterns.
  • High-level: Extracting linguistic meaning and emotional valence.

To ensure the model understands the broader context, contrastive learning is used to align these audio features with text transcripts processed via BERT. Simultaneously, a Visual Encoder extracts 3D Morphable Model (3DMM) coefficients from the speaker’s video to provide additional context regarding the speaker's own expressions and pose.

2. Sequential Motion Decoding

The fused audio and visual features are fed into a six-layer bi-directional Gated Recurrent Unit (GRU). This sequential decoder acts as the system's "brain," predicting the listener’s 3DMM coefficients over time. The bi-directional nature of the GRU allows the model to maintain temporal consistency, ensuring that reactions flow naturally rather than appearing as a series of disconnected frames.

3. 3D-Aware Rendering

A major bottleneck in previous listener models was the rendering phase, which often produced artifacts during sharp head turns. This pipeline utilizes 3DFaceShop, a 3D-aware portrait model. Unlike standard 2D-based renderers, 3DFaceShop maintains a stable 3D representation of the head, significantly improving background stability and identity preservation even during complex rotations.

4. Video Restoration

To achieve photo-realistic results, the pipeline includes a final enhancement step using the Recurrent Video Restoration Transformer (RVRT). This step sharpens facial details and mitigates the low-resolution limitations of the training data, resulting in a high-definition output suitable for modern display standards.

Experimental Results

The model was evaluated using the extended ViCo dataset, comprising 440 training pairs and 45 test clips. The pipeline dominated the ViCo@2023 official leaderboard, taking first place overall and winning five out of nine primary evaluation metrics.

Quantitative Performance

The system showed significant improvements over the ViCo baseline across several key metrics:

  • Visual Quality: The model achieved a Structural Similarity Index (SSIM) of 0.647 and a Peak Signal-to-Noise Ratio (PSNR) of 26.757, compared to the baseline's 0.495 and 15.335.
  • Motion Accuracy: The pipeline led the field in PoseL1 (head position accuracy) and ExpL1 (expression accuracy), indicating a high degree of correlation between the speaker's cues and the listener's movements.
  • Realism: Ablation studies demonstrated that the full pipeline reduced the Fréchet Inception Distance (FID)—a standard measure of image realism—from 41.3 to 30.6.

Discussion and Limitations

The success of this pipeline highlights two critical insights: first, that semantic understanding of audio must be layered to capture the full spectrum of human reaction; and second, that high-quality rendering and restoration are as vital as the underlying motion prediction.

However, limitations remain. The relatively small size of the training dataset (440 pairs) restricts the variety of behavioral styles the model can emulate. Additionally, while the model produces appropriate reactions, it does not yet allow for "style personalization," where a user could specify a listener's unique personality (e.g., an active nodder versus a stoic listener). Future work will focus on expanding the dataset and developing methods for more granular behavioral control.

Conclusion

By integrating hierarchical semantic features with advanced 3D rendering and restoration techniques, this pipeline sets a new benchmark for responsive listener head generation. The results from the ViCo@2023 challenge suggest that this multi-stage approach is highly effective for creating realistic, context-aware digital humans in conversational settings.

Summary updated

BibTeX
@inproceedings{chang2023hierarchical,
  author = {Zhigang Chang and Weitai Hu and Qing Yang and Shibao Zheng},
  title = {{Hierarchical Semantic Perceptual Listener Head Video Generation: A High-performance Pipeline}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612869},
}
Figure