Speech-driven facial animation using polynomial fusion of features

Triantafyllos Kefalas, Konstantinos Vougioukas, Yannis Panagakis, Stavros Petridis, Jean Kossaifi, Maja PanticOpenReview 2019 · DOI · OpenReview

Abstract

Speech-driven facial animation involves using a speech signal to generate realistic videos of talking faces. Recent deep learning approaches to facial synthesis rely on extracting low-dimensional representations and concatenating them, followed by a decoding step of the concatenated vector. This accounts for only first-order interactions of the features and ignores higher-order interactions. In this paper we propose a polynomial fusion layer that models the joint representation of the encodings by a higher-order polynomial, with the parameters modelled by a tensor decomposition. We demonstrate the suitability of this approach through experiments on generated videos evaluated on a range of metrics on video quality, audiovisual synchronisation and generation of blinks.

Introduction

Generating realistic video of a person speaking based solely on an audio track is a significant challenge in computer graphics and computer vision. It requires the precise coordination of facial muscles to ensure accurate lip-sync while maintaining natural non-verbal cues, such as eye blinks and eyebrow movements.

Most contemporary deep learning models address this by extracting separate latent representations for the speaker's identity and the audio signal, which are then concatenated before being fed into a video generator. However, simple concatenation only captures first-order interactions, treating audio and identity as independent components. This often results in synthetic videos that are sensitive to large facial movements or fail to capture the nuanced, person-specific relationship between speech and facial dynamics.

Proposed Methodology

The core contribution of this work is the Polynomial Fusion Layer. Instead of simple vector concatenation, the joint representation of audio and identity is modeled as a higher-order polynomial. This allows the network to learn complex interactions between specific audio features and facial characteristics.

The proposed architecture featuring the Polynomial Fusion Layer, which integrates identity, audio, and noise encodings to drive a U-Net based generator.

Tensor Decomposition for Efficient Fusion

Modeling all possible interactions between high-dimensional vectors would typically lead to exponential parameter growth. To mitigate this, the authors employ low-rank tensor factorizations, specifically Canonical Polyadic (CP) and Tucker decompositions. These techniques compress the complex mathematical interactions into a manageable number of parameters without sacrificing the ability to model higher-order relationships.

System Architecture

The system utilizes an encoder-decoder Generative Adversarial Network (GAN) structure:

  • Encoders: Separate networks process the identity (a single face image), audio (0.2s segments), and noise (to introduce natural variability like blinks).
  • Generator: A U-Net based architecture receives the fused features and generates the corresponding video frames.
  • Discriminators: Three distinct critics evaluate the frames for image quality, sequence plausibility, and audiovisual synchronization.

Experimental Results

The model was evaluated on three major audiovisual datasets: GRID (structured sentences), TCD-TIMIT (extensive sentence variety), and CREMA-D (diverse emotional expressions).

Quantitative comparison of the proposed polynomial fusion variants (PF-CP, PF-CMF, etc.) against state-of-the-art methods across multiple metrics and datasets.

Performance Metrics

The proposed polynomial fusion models (notably the PF-CP and PF-CMF variants) consistently outperformed or matched state-of-the-art baselines like Speech2Vid and Vougioukas et al. (2019):

  • Image Quality: Higher PSNR and SSIM scores across all datasets indicate clearer and more accurate frame generation.
  • Synchronization: Superior performance in SyncNet evaluations (AV confidence) demonstrates more accurately timed lip movements.
  • Identity Preservation: Average Content Distance (ACD) metrics show that the fusion method better maintains the subject's identity throughout the sequence.
  • Natural Spontaneity: By incorporating a temporal noise encoder, the model generates spontaneous blinks at a rate of approximately 0.28 per second, closely mimicking human behavior.

Conclusion and Future Work

This research demonstrates that moving beyond simple feature concatenation to higher-order polynomial fusion significantly enhances the realism of speech-driven facial animation. While the current model handles spontaneous cues like blinks effectively, future work could explore end-to-end training of encoders alongside the fusion layer. Additionally, extending the model to handle extreme emotional transitions and varied head poses remains an important area for further development.

Summary updated

Abstract

Speech-driven facial animation involves using a speech signal to generate realistic videos of talking faces. Recent deep learning approaches to facial synthesis rely on extracting low-dimensional representations and concatenating them, followed by a decoding step of the concatenated vector. This accounts for only first-order interactions of the features and ignores higher-order interactions. In this paper we propose a polynomial fusion layer that models the joint representation of the encodings by a higher-order polynomial, with the parameters modelled by a tensor decomposition. We demonstrate the suitability of this approach through experiments on generated videos evaluated on a range of metrics on video quality, audiovisual synchronisation and generation of blinks.
BibTeX
@misc{kefalas2019speech,
  author = {Triantafyllos Kefalas and Konstantinos Vougioukas and Yannis Panagakis and Stavros Petridis and Jean Kossaifi and Maja Pantic},
  title = {{Speech-driven facial animation using polynomial fusion of features}},
  year = {2019},
  eprint = {1912.05833},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.05833},
  howpublished = {OpenReview},
}
Figure