From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech

Hyeong-Seok Choi, Changdae Park, Kyogu Lee

Can you "see" a person just by hearing them? Neuroscience suggests that humans naturally link vocal characteristics to facial structures because both are influenced by the physical shape of the vocal tract. In machine learning, however, most "speech-to-face" models rely on massive datasets labeled with identities, genders, or nationalities.

Researchers have developed a novel framework that generates a human face image from a short audio clip of their speech without ever using human-labeled identity tags. By linking a self-supervised identity-matching stage with a conditional GAN, the system learns to "imagine" a face that captures the speaker's likely age, gender, and ethnicity directly from the raw audio waveform.

The Challenge of Unlabeled Cross-Modal Learning

The authors of this paper ask a fundamental question: Is it possible to learn this cross-modal relationship using only "in-the-wild" video data without any human annotations?

The challenge lies in the fact that a voice doesn't contain all the information needed to reconstruct a face (like a specific haircut, lighting, or head pose), and the differences between speakers can be incredibly subtle for a neural network to distinguish without explicit labels.

A Two-Stage, End-to-End Framework

To solve this, the authors propose a two-stage, end-to-end framework that bridges inference and generation.

The proposed two-stage framework: Inference and Generation.

1. The Inference Stage

The model first learns to match voices to faces. It uses a SincNet (which processes raw audio waveforms) and a 2D-CNN face encoder. These are trained using a self-supervised "cross-modal identity matching" task: given a voice, pick the correct face out of $K$ options. No labels are needed; the model simply learns that the face and voice appearing in the same video clip belong together.

2. The Generation Stage

The trained encoders are then plugged into a Conditional Generative Adversarial Network (cGAN).

  • Disentangled Latents: The generator takes two inputs: a deterministic speech embedding ($c$) and a random noise vector ($z$). This allows the model to separate factors related to the voice (identity, age) from irrelevant factors (lighting, head orientation).
  • RelidGANs Loss: To help the model distinguish between very similar speakers, the authors introduced a "relativistic identity cGAN" loss. This forces the discriminator to penalize mismatched face-voice pairs more aggressively, leading to more distinct generated identities.

Key Insights

  • The Necessity of the Two-Stage Approach: One of the most striking findings was that skipping the inference (matching) stage caused the generator to completely ignore the speech signal. Without the pre-trained "understanding" of identity, the GAN simply generated random faces.
  • Disentanglement is Natural: By providing a random noise vector ($z$) alongside the speech condition ($c$), the model naturally learned to map pose and background to the noise, while keeping the "human" characteristics tied to the audio.
  • Raw Waveforms Matter: Using SincNet to process raw audio (rather than standard spectrograms) allowed the model to capture finer biometric details of the speaker's voice that might be lost in traditional feature extraction.

Datasets and Key Results

The model was trained primarily on AVSpeech, a massive "in-the-wild" dataset containing 2.8 million YouTube clips (using 800k clips for training). This dataset is challenging because it includes diverse lighting, expressions, and occlusions. VoxCeleb and VGGFace were used primarily for comparison and quantitative analysis.

Despite being fully self-supervised, the model achieved impressive results:

  • Identity Matching: The model achieved 89.22% accuracy on a 2-way face-from-voice matching task, performing competitively with models trained on labeled data.
  • Retrieval Performance: In a "Face Image Retrieval" test (using a generated face to find the real person in a gallery of 5,000), the proposed model outperformed previous state-of-the-art supervised methods like Speech2Face.
  • Qualitative Success: The generated images show smooth transitions when interpolating between voices, accurately shifting features like gender, age, and ethnicity while maintaining a consistent "style" when the random noise vector is held constant.
Interpolation between different voices showing smooth transitions in generated facial features.

Limitations and Future Work

The authors acknowledge that it is physically impossible to perfectly reconstruct a face from a voice alone—factors like glasses, specific hairstyles, or scars aren't encoded in audio.

Furthermore, the paper highlights a significant data bias problem. Because the model learns from YouTube data, it inherits the demographic biases of that platform. Future work aims to address these biases (ethnicity, gender, age) through better sampling strategies and to expand the framework to other multi-modal tasks beyond just faces and voices.

Summary updated

Abstract

This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations. To this end, we propose a multi-modal learning framework that links the inference stage and generation stage. First, the inference networks are trained to match the speaker identity between the two different modalities. Then the pre-trained inference networks cooperate with the generation network by giving conditional information about the voice. The proposed method exploits the recent development of GANs techniques and generates the human face directly from the speech waveform making our system fully end-to-end. We analyze the extent to which the network can naturally disentangle two latent factors that contribute to the generation of a face image one that comes directly from a speech signal and the other that is not related to it and explore whether the network can learn to generate natural human face image distribution by modeling these factors. Experimental results show that the proposed network can not only match the relationship between the human face and speech, but can also generate the high-quality human face sample conditioned on its speech. Finally, the correlation between the generated face and the corresponding speech is quantitatively measured to analyze the relationship between the two modalities.
BibTeX
@inproceedings{choi2020from,
  author = {Hyeong-Seok Choi and Changdae Park and Kyogu Lee},
  title = {{From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=7h9WVKkC_Il},
}
Figure