3D Face Synthesis Driven by Personality Impression

Yining Lang, Wei Liang, Yujia Wang, Lap-Fai YuAAAI 2019 · DOI

Abstract

Creating 3D characters that evoke specific personality impressions—such as "smart," "hostile," or "confident"—is a critical yet labor-intensive task in digital media. This paper introduces a data-driven framework that automates this process by combining deep learning classifiers with a Markov Chain Monte Carlo (MCMC) optimization process. The system subtly modifies 3D facial geometry and texture to match a target trait while strictly maintaining the original character's identity.

Introduction

In game development, animation, and virtual reality, designers often need to create characters that "look the part." A hero should look confident, while a villain might need to look hostile. Currently, this is a tedious, manual trial-and-error process where artists modify meshes and iterate based on subjective feedback.

The challenge lies in the fact that personality impressions are driven by incredibly subtle facial details that are difficult to define mathematically. Furthermore, creating a face that is, for example, "friendly but silly" is significantly more complex than simply making a face look "happy." This creates a scalability bottleneck when a project requires hundreds of unique, expressive non-player characters (NPCs).

Methodology

The authors propose a two-stage pipeline consisting of a Learning Stage and an Optimization Stage.

1. The Learning Stage

In this initial phase, the system learns to recognize personality traits and identify preservation constraints.

  • Personality Classification: The researchers fine-tuned a GoogLeNet architecture on a dataset of 10,000 2D face images annotated with personality labels. This allows the model to output a quantitative "personality score" for any given face.
  • Face Similarity Metric: To ensure the optimization doesn't transform a character into a completely different person, a Siamese Network was trained on the Labeled Faces in the Wild (LFW) dataset. This network calculates the similarity between faces, acting as a constraint during synthesis.
The learning pipeline utilizes a fine-tuned GoogLeNet for personality classification and a Siamese Network for identity preservation.

2. The Optimization Stage

The system takes an input 3D face (represented using a multi-linear PCA model from the Basel Face Model) and a target personality trait. It then employs a data-driven MCMC sampler to iteratively modify the face's geometry and texture.

  • Region-Move: This novel approach mimics human intuition. By analyzing how professional artists modify faces, the system learns which regions (eyes, jaw, mouth, etc.) are most impactful for specific traits and prioritizes modifications there.
  • Prior-Move: The sampler uses a prior distribution learned from the personality dataset to suggest global changes likely to induce the desired impression.
The optimization process uses an MCMC sampler to iteratively modify facial regions and evaluate them against personality and similarity metrics.

The final cost function balances two primary objectives: maximizing the target personality score and minimizing the deviation from the original identity.

Key Insights and Results

Subtlety and Artist Mimicry

The paper demonstrates that personality perception is not just about "large" features like nose size, but rather the configuration of subtle details that convolutional neural networks (CNNs) are uniquely suited to identify. By using the "Region-Move" strategy, the algorithm avoids a blind mathematical search and instead focuses on the facial areas that human artists naturally prioritize.

Expression Invariance

A significant finding is that personality impressions are relatively "expression-invariant." A face optimized to look "confident" maintains that impression even when forced into different expressions, such as anger or happiness. This suggests that the underlying geometry of personality is distinct from the muscle movements associated with emotion.

Performance Metrics

In a perceptual validation study with 160 participants, the average recognition accuracy for synthesized traits was 38%. While this may seem modest, it is more than triple the 12.5% chance-level for an eight-category selection. Traits like "Friendly" and "Hostile" were the most recognizable (45%–50% accuracy), whereas "Humorous" and "Boring" proved more difficult to distinguish. The optimization is efficient, taking approximately 15 seconds (100–150 iterations) on standard consumer hardware.

Limitations and Future Work

The current model focuses exclusively on the "naked" face, excluding hair, accessories, and skin wrinkles, which significantly influence personality perception. Additionally, the study is limited to static meshes and does not account for the impact of movement or speech. Future work may also address cultural biases in personality perception, as the current model is trained on a dataset that may not generalize across all global cultures.

Summary updated

BibTeX
@inproceedings{lang2019d,
  author = {Yining Lang and Wei Liang and Yujia Wang and Lap-Fai Yu},
  title = {{3D Face Synthesis Driven by Personality Impression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33011707},
}
Figure