Abstract
This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations. To this end, we propose a multi-modal learning framework that links the inference stage and generation stage. First, the inference networks are trained to match the speaker identity between the two different modalities. Then the trained inference networks cooperate with the generation network by giving conditional information about the voice. The proposed method exploits the recent development of GANs techniques and generates the human face directly from the speech waveform making our system fully end-to-end. We analyze the extent to which the network can naturally disentangle two latent factors that contribute to the generation of a face image - one that comes directly from a speech signal and the other that is not related to it - and explore whether the network can learn to generate natural human face image distribution by modeling these factors. Experimental results show that the proposed network can not only match the relationship between the human face and speech, but can also generate the high-quality human face sample conditioned on its speech. Finally, the correlation between the generated face and the corresponding speech is quantitatively measured to analyze the relationship between the two modalities.
Introduction
Humans possess a natural ability to associate vocal characteristics with physical traits. While we cannot perfectly reconstruct a photograph from a soundbite, we often infer age, gender, and even facial structure from a person's voice. This paper explores the possibility of automating this process through a fully self-supervised, end-to-end framework that generates human faces from raw speech waveforms without any human-labeled annotations.
Problem Statement
Generating a face from a voice is inherently a "one-to-many" problem. A single voice can belong to an individual in various lighting conditions, poses, or environments. Previous research often relied on supervised learning with explicit identity labels or pre-trained face recognition models. This work addresses three core challenges: achieving self-supervision using "in-the-wild" data, disentangling voice-correlated traits (identity) from unrelated factors (pose, lighting), and maintaining an end-to-end pipeline from raw audio to high-resolution images.
Methodology
The proposed framework consists of two distinct but linked stages: the inference stage and the generation stage.
The Inference Stage
The model first learns to match voices to faces through a cross-modal identity matching task. Two encoders are employed: a Speech Encoder utilizing SincNet to process raw waveforms and a Face Encoder based on a 2D-CNN with residual connections. These encoders map both modalities into a shared embedding space. By training the model to identify the correct face-voice pair among $K$ candidates, the system learns to extract identity as the common denominator between audio and visual data.
The Generation Stage
The pre-trained speech encoder is then integrated into a Conditional GAN (cGAN). The generator receives a deterministic speech embedding ($c$) and a random noise vector ($z$). Using Adaptive Instance Normalization (AdaIN), the voice information is injected into the image generation process. To ensure identity consistency, the authors introduced a Relativistic Identity cGAN (relidGANs) loss, which penalizes the discriminator if it fails to distinguish between correctly paired and mismatched face-voice samples.
Experimental Setup and Datasets
The model was primarily trained on a subset of the AVSpeech dataset, containing 800,000 "in-the-wild" YouTube clips. This dataset lacks identity labels, serving as a rigorous test for the self-supervised approach. For benchmarking, the researchers also utilized VoxCeleb and VGGFace, which provide identity labels for public figures.
Results and Discussion
The framework demonstrated significant capabilities in both identity matching and face generation.
Disentanglement of Latent Factors
A key finding was the model's ability to naturally disentangle latent factors. The speech embedding ($c$) effectively controls identity-related traits such as gender, ethnicity, and age. Conversely, the stochastic noise vector ($z$) captures environmental factors like head orientation, lighting, and background.
Quantitative Performance
On the AVSpeech dataset, the self-supervised inference model achieved 54.33% top-1 accuracy in a 10-way matching task, far exceeding the 10% random chance baseline. In face retrieval tasks, the model outperformed the previous state-of-the-art (Speech2Face) across metrics like Cosine Distance (CD) and $L_1/L_2$ distances. Furthermore, quantitative analysis confirmed a high correlation between generated face embeddings and original speech, indicating that the model successfully conditions generation on vocal input rather than producing arbitrary faces.
Conclusion and Future Work
This work presents a robust end-to-end system for generating faces from speech without manual labels. While the model achieves impressive results, limitations remain. Perfect reconstruction is impossible as some physical traits (e.g., specific hairstyles or glasses) are not encoded in the voice. Additionally, the system is subject to demographic biases present in the training data. Future research will focus on unsupervised data sampling to mitigate bias and increasing the resolution of generated images beyond the current 128x128 standard.
Summary updated