Neutral Face Game Character Auto-Creation via PokerFace-GAN

Tianyang Shi, Zhengxia Zou, Xinhui Song, Zheng Song, Changjian Gu, Changjie Fan, Yi Yuan

The Challenge of Character Customization

In modern Role-Playing Games (RPGs), character customization is incredibly deep, often involving hundreds of manual parameters. While "Face-to-Parameter" (F2P) AI methods have emerged to automate this process, they frequently suffer from a major flaw known as parameter coupling.

Manual eyebrow parameter sliders in a game engine illustrate the complexity of traditional character customization.

If a user uploads a photo of themselves smiling, previous AI methods might "bake" that smile into the character's base identity. In the game's engine, the character might end up with a permanent, slightly creepy grin because the AI could not distinguish between actual bone structure and the temporary movement of facial muscles. Furthermore, many existing methods use "neural renderers" that only work well for front-facing photos, making them difficult to use for different poses or lighting conditions.

Methodology: Disentangling Identity and Expression

The authors propose PokerFace-GAN, a framework that treats character creation as a "source separation" problem. Just as an audio filter might separate vocals from background music, PokerFace-GAN separates the "static" face (identity) from the "dynamic" face (expression). This is achieved through a multi-task self-supervised learning approach that minimizes several losses simultaneously: identity similarity (using a face recognition net), facial content (using segmentation), landmarks, and a "loop-back" loss to ensure the predictor can correctly interpret its own generated images. The system consists of three main components:

  1. A Differentiable Character Renderer ($R$): Unlike previous "neural" renderers, this is a "hard-programmed" renderer that mimics a real game engine (like Unity) using a bone-driven model. Because it is differentiable, the system can calculate how changing a 3D parameter affects the final 2D image pixels, allowing the model to learn through backpropagation.
  2. The Predictor ($P$): This network takes an input photo and translates it into three groups of parameters: identity (bone structure), expression (muscle movement), and pose (head orientation).
  3. The PokerFace Discriminator ($D$): This is the "GAN" part of the name. The authors created a "poker face pool" of neutral-expression images. The discriminator is trained to distinguish between the predicted identity parameters and these "true" neutral parameters. This forces the Predictor to strip all expression data out of the identity parameters and shove it into the expression category.

Key Results and Performance

The model was trained on the CelebA dataset and evaluated using seven major face verification benchmarks (LFW, CFP_FF, CFP_FP, AgeDB, CALFW, CPLFW, and Vggface2_FP). PokerFace-GAN outperformed previous state-of-the-art methods across all datasets. For example, on the LFW benchmark, it achieved a score of 0.9550, compared to 0.9402 for the previous best method.

A generated 3D character showing the neutral 'poker face' identity isolated from the input expression.

Beyond accuracy, the system is highly efficient, running at approximately 600Hz on a GTX 1080Ti, making it fast enough for real-time use in a game's UI. Visual results show that even when input photos have open mouths or squinted eyes, the "identity-only" output is a clean, neutral 3D model that still clearly resembles the user. By building a renderer that actually uses bones and meshes (rather than just a neural network that looks like a renderer), the authors made the system much more scalable, allowing it to handle multi-view angles and different poses that would break a standard neural renderer.

Limitations and Future Work

While effective, the system currently struggles when a user's eyes are completely closed in the input photo, largely due to inaccuracies in the landmark extraction tools used during training. Additionally, the model can struggle with very complex or "mixed" expressions, likely because the CelebA training data mostly features neutral faces or simple smiles. Future work could involve training on larger, more diverse datasets to improve the AI's ability to handle a wider variety of human expressions and ethnicities. This would further enhance the model's robustness and ensure its applicability across a broader spectrum of global users.

Summary updated

BibTeX
@inproceedings{shi2020neutral,
  author = {Tianyang Shi and Zhengxia Zou and Xinhui Song and Zheng Song and Changjian Gu and Changjie Fan and Yi Yuan},
  title = {{Neutral Face Game Character Auto-Creation via PokerFace-GAN}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=43mY4WtH09o},
}
Figure