Faces à la Carte: Text-to-Face Generation via Attribute Disentanglement

Tianren Wang, Teng Zhang, Brian C. Lovell

TL;DR

This paper introduces TTF-HD, a novel Text-to-Face generation model that produces high-resolution (1024x1024) and diverse facial images from free-form text descriptions. It addresses the challenge of complex facial attribute descriptions by disentangling features and allowing for controlled manipulation of latent space vectors.

What Problem Does This Paper Address?

Generating realistic images from text descriptions (Text-to-Image synthesis) is a rapidly advancing field. However, generating faces from text (Text-to-Face, TTF) presents unique challenges. Facial descriptions can be far more nuanced and detailed than those for objects like birds or flowers. Furthermore, the relationship between textual attributes and facial features is less direct, making it harder to achieve precise control. Existing TTF methods often struggle with image quality, text-image consistency, and generating diverse outputs for the same description. This is particularly relevant for applications like criminal investigations (sketching suspects from witness descriptions) and addressing data scarcity in biometrics research while respecting ethical concerns around data consent.

Methods

The TTF-HD model is built upon several key components:

  • Multi-Label Text Classifier (T): A fine-tuned BERT model is used to process natural language descriptions and output a 40-dimensional vector representing facial attributes. This classifier can handle descriptions of varying lengths, unlike previous methods that were limited to short sentences.
  • Image Encoder (E): A fine-tuned MobileNet model acts as an image encoder, predicting feature labels from generated images. This helps in understanding the latent space representation of facial attributes.
  • Feature Disentanglement and Manipulation: The core innovation lies in how the model manipulates the latent space of a pre-trained generator.
  • Feature Axes: The paper defines 40 orthogonal "feature axes" in the latent space, each corresponding to a facial attribute. These axes are learned by finding the relationship between noise vectors and predicted feature labels.
  • Differentiated Embeddings: To enable movement in both positive and negative directions along these axes (e.g., "young" vs. "old"), the model computes "differentiated embeddings" by subtracting the predicted image embedding from the target text embedding. This allows for finer control.
  • Nonlinear Reweighting: To emphasize specified features and minimize interference from unspecified ones, a nonlinear reweighting mechanism is applied to the differentiated embeddings.
  • Noise Vector Manipulation: The noise vector is then adjusted along these feature axes based on the reweighted differentiated embeddings. This process is repeated for each specified feature.
  • High-Resolution Generator (G): A pre-trained StyleGAN2 model is used as the generator, capable of producing high-resolution (1024x1024) images. The manipulated noise vectors are fed into StyleGAN2 to synthesize faces with the desired attributes.

Datasets

  • CelebA [14]: This dataset, containing over 200,000 face images with 40 paired one-shot label vectors, was used for training the multi-label text classifier and the image encoder.
  • Text Description Corpus: A paired text description corpus, where each description has up to 10 sentences, was also used for training.

Key Insights and Ideas

  • Complex Descriptions are Key: The paper highlights that facial descriptions are inherently more complex than for other objects, necessitating a robust text encoder capable of handling longer, more nuanced inputs.
  • Disentanglement for Control: The concept of learning orthogonal "feature axes" in the latent space is crucial for disentangling facial attributes, allowing for independent manipulation of features like age, hair color, or expression.
  • Differentiated Embeddings for Bidirectional Control: Moving beyond simple positive attribute prediction, the use of differentiated embeddings (target - predicted) is a clever way to enable bidirectional control along feature axes, addressing limitations of one-hot encoded or single-value attributes.
  • Diversity is Essential: The paper emphasizes that for TTF, generating multiple diverse faces from a single description is vital, mimicking how different people might describe the same individual or providing options for applications like witness selection.

Key Results

  • Qualitative: The generated images exhibit good image quality and accurately reflect the specified facial features from the text descriptions. Figure 1 and Figure 4 showcase examples of synthesized faces with varying attributes.
  • Quantitative:
    • The TTF-HD model outperforms the state-of-the-art AttnGAN [21] in terms of both image quality (Inception Score - IS) and text-to-image similarity (Cosine Similarity - CS).
  • Specifically, TTF-HD achieved an IS of 1.117±0.127 and a CS of 0.664, compared to AttnGAN's 1.062±0.051 and 0.511 respectively.
  • The model also demonstrates high image diversity, as shown in Figure 4, where a single sentence description yields varied facial features.
  • Ablation Study: The ablation study (Table 2 and Figure 5) demonstrates that applying all four proposed operations (reweighting, differentiation, normalization, and feature lock) leads to the best trade-off between image quality, text-to-face similarity, and diversity. Removing operations results in a degradation of these metrics and can lead to artefacts or mode collapse.

Limitations and Future Work

  • Robustness: The authors acknowledge that the model is not entirely robust, with occasional images in a batch being less consistent with the text descriptions. This is attributed to potential inaccuracies in the text classifier and image encoder, possibly due to insufficient training data.
  • Feature Correlation: Features in the latent space are not perfectly disentangled, meaning manipulating one feature axis might unintentionally alter highly correlated features.
  • Future Directions: Addressing the robustness issues and improving feature disentanglement are identified as key areas for future research.

Summary updated

Abstract

Text-to-Face (TTF) synthesis is a challenging task with great potential for diverse computer vision applications. Compared to Text-to-Image (TTI) synthesis tasks, the textual description of faces can be much more complicated and detailed due to the variety of facial attributes and the parsing of high dimensional abstract natural language. In this paper, we propose a Text-to-Face model that not only produces images in high resolution (1024×1024) with text-to-image consistency, but also outputs multiple diverse faces to cover a wide range of unspecified facial features in a natural way. By fine-tuning the multi-label classifier and im age encoder, our model obtains the adjustment vectors and image embeddings which are used to transform the input noise vector sampled from the normal distribution. Afterwards, the transformed noise vector is fed into a pre-trained high-resolution image generator to produce a set of faces with the desired facial attributes. We refer to our model as TTF-HD. Experimental results show that TTF-HD generates high-quality synthesised faces from free-form text descriptions with state-of-the-art performance.
BibTeX
@inproceedings{wang2021faces,
  author = {Tianren Wang and Teng Zhang and Brian C. Lovell},
  title = {{Faces à la Carte: Text-to-Face Generation via Attribute Disentanglement}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=-SCvjsOwkdh},
}
Figure