Audiovisual Speaker Conversion: Jointly and Simultaneously Transforming Facial Expression and Acoustic Characteristics

Fuming Fang, Xin Wang, Junichi Yamagishi, Isao Echizen

TL;DR

This paper introduces an Audiovisual Speaker Conversion (AVSC) system that jointly and simultaneously transforms both the voice and facial expressions of a source speaker into those of a target speaker. By fusing acoustic and visual features during the transformation process, the model preserves the natural correlation between speech and facial movements, resulting in significantly better synchronization and naturalness compared to transforming audio and video independently.

What Problem Does This Paper Address?

Voice conversion (changing a person's voice) and face transformation (like deepfakes) are increasingly common technologies used in gaming, privacy protection, and film. However, when you want to change both the voice and the face of a speaker, the standard approach is to run two separate algorithms.

This disconnected approach often leads to a loss of naturalness. Because the audio and video models don't communicate, the resulting video suffers from asynchronous lip movements, mismatched emotional expressions, and timing errors. The authors tackle this by asking: can we build a unified model that explicitly learns the correlation between acoustic characteristics and facial movements to transform them together?

Methods

The authors propose a joint transformation pipeline consisting of three main neural network components:

  1. Audiovisual Transformation Network: This is the core novelty. It takes source acoustic features (mel-spectrograms) and facial features (VGG-19 embeddings combined with OpenPose facial keypoints). Because video (25 fps) and audio (sampled every 5 ms) operate at different rates, the network uses 1D Convolutional Neural Networks (CNNs) with specific strides to align their sampling rates. It fuses these modalities into a shared representation, transforms them into the target speaker's domain, and then up-samples/down-samples them back to their respective frame rates.
  2. WaveNet Vocoder: An autoregressive waveform generator adapted to take both the converted mel-spectrogram and the converted facial features as conditioning inputs to synthesize the final target audio.
  3. Image Reconstruction Network: A Generative Adversarial Network (specifically, LSGAN) that synthesizes the final RGB video frames from the fused, converted acoustic and facial features.

To prove the value of this joint approach, the authors compared it against a baseline method that used the exact same architecture but separated the acoustic and facial transformation paths.

Datasets

Because this task requires highly synchronized emotional data, the authors created a custom Emotional Audiovisual Database:

  • Subjects: Two Japanese female actors (one assigned as the source, the other as the target).
  • Content: 700 parallel sentences per actor.
  • Emotions: 7 distinct emotional states (neutral, normal/strong happiness, normal/strong sadness, normal/strong anger), with 100 sentences per emotion.
  • Splits: 630 samples were used for training and 70 for testing.

Key Insights and Ideas

  • Modality Synergy: Jointly transforming audio and video forces the network to maintain the natural correlation between lip movements and speech sounds. The modalities actually help compensate for one another; for example, facial expressions provide strong cues for emotional state, which helps the network generate more emotionally accurate speech characteristics.
  • Facial Feature Dominance: The authors observed that facial features tend to dominate the fused latent representation. While this is excellent for ensuring visual synchronization and accurate lip movements, it makes it slightly difficult for the network to fully leverage acoustic features for visual generation.
  • Emotion is Harder than Neutral: Both the baseline and the proposed method scored lower on emotional test samples compared to neutral ones. However, the proposed fused method closed this gap significantly, indicating that tying facial movements to speech is crucial for convincing emotional deepfakes.

Key Results

The proposed joint method outperformed the separate-transformation baseline across both objective and subjective metrics:

  • Objective Correlation: Canonical correlation analysis showed that the proposed method achieved a noticeably higher correlation coefficient ($r$) between the generated mel-spectrograms and lip movements compared to the baseline.
  • Subjective Naturalness (MOS): In a 1-to-5 Mean Opinion Score (MOS) test, human evaluators rated the proposed method higher across all modalities:
  • Audiovisual: 3.26 (Proposed) vs. 2.95 (Baseline)
  • Audio-only: 2.40 (Proposed) vs. 2.09 (Baseline)
  • Visual-only: 3.47 (Proposed) vs. 3.43 (Baseline)
  • Speaker Similarity: In A/B preference tests, evaluators strongly preferred the proposed method for sounding and looking more like the target speaker, likely because the joint identity features helped the network estimate more accurate parameters.

Limitations and Future Work

  • Feature Imbalance: As noted, the facial features dominate the transformation. The authors plan to improve the architecture to better balance the influence of acoustic and facial features.
  • Reliance on Parallel Data: The current system requires strictly parallel training data—meaning the source and target speakers must utter the exact same sentences so they can be aligned via Dynamic Time Warping (DTW).
  • Future Work: To make the system more practical for real-world use, the authors plan to explore non-parallel training methods (such as cycle-consistent adversarial networks) so the model can learn from unaligned, independent videos of the source and target speakers.

Summary updated

Abstract

An audiovisual speaker conversion method is presented for simultaneously transforming the facial expressions and voice of a source speaker into those of a target speaker. Transforming the facial and acoustic features together makes it possible for the converted voice and facial expressions to be highly correlated and for the generated target speaker to appear and sound natural. It uses three neural networks: a conversion network that fuses and transforms the facial and acoustic features, a waveform generation network that produces the waveform from both the converted facial and acoustic features, and an image reconstruction network that outputs an RGB facial image also based on both the converted features. The results of experiments using an emotional audiovisual database showed that the proposed method achieved significantly higher naturalness compared with one that separately transformed acoustic and facial features.
BibTeX
@inproceedings{fang2019audiovisual,
  author = {Fuming Fang and Xin Wang and Junichi Yamagishi and Isao Echizen},
  title = {{Audiovisual Speaker Conversion: Jointly and Simultaneously Transforming Facial Expression and Acoustic Characteristics}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8683872},
}
Figure