CG-GAN: An Interactive Evolutionary GAN-based Approach for Facial Composite Generation)

Nicola Zaltron, Luisa Zurlo, Sebastian Risi

Abstract

Facial composites are graphical representations of an eyewitness's memory of a face. Many digital systems are available for the creation of such composites but are either unable to reproduce features unless previously designed or do not allow holistic changes to the image. In this paper, we improve the efficiency of composite creation by removing the reliance on expert knowledge and letting the system learn to represent faces from examples. The novel approach, Composite Generating GAN (CG-GAN), applies generative and evolutionary computation to allow casual users to easily create facial composites. Specifically, CG-GAN utilizes the generator network of a pg-GAN to create high-resolution human faces. Users are provided with several functions to interactively breed and edit faces. CG-GAN offers a novel way of generating and handling static and animated photo-realistic facial composites, with the possibility of combining multiple representations of the same perpetrator, generated by different eyewitnesses.

Introduction

Traditional facial composites—the "police sketches" used in criminal investigations—face a major bottleneck: the human element. Historically, these required a forensic artist to translate a witness's verbal description into a drawing. Digital alternatives often rely on "feature-based" systems (picking a nose from a list), which fail to capture the holistic way humans actually remember faces.

Recent AI approaches using Latent Variable Evolution (LVE) allowed users to "evolve" faces by selecting the best images from a lineup. However, these systems suffered from a lack of control: if you liked the eyes but hated the chin, a "mutation" to fix the chin would likely scramble the eyes too. This leads to "user fatigue," where the witness gives up before reaching a likeness. CG-GAN aims to bridge the gap between holistic recognition and granular, constructive control.

Methods

The researchers built CG-GAN (Composite Generating GAN) by stacking several state-of-the-art techniques:

  1. The Engine: They used a pg-GAN (Progressive Growing of GANs), which is capable of generating high-resolution, photorealistic human faces.
  2. The Search: They employed Latent Variable Evolution (LVE). Instead of manually editing pixels, the system searches the "latent space" (the mathematical representation of faces) using evolutionary algorithms. Users select "parent" faces to produce "offspring" for the next generation.
  3. The Control (Feature Extraction): Using the TL-GAN approach, the team identified specific "axes" in the latent space that correspond to traits like "beard," "pointy nose," or "pale skin."
  4. The Innovation (Smart Locks): This is the paper's core contribution. To prevent the "scrambling" effect, they implemented Feature Locks. When a user likes a specific trait, they can lock it. The system then orthogonalizes the remaining axes, ensuring that changes to other features don't interfere with the locked ones. Smart Locks take this further by automatically locking highly correlated features (e.g., locking "gender" might automatically lock "beard" and "makeup" based on a cosine similarity threshold of ±0.5).
Examples of high-resolution facial composites generated by the system.

The model was pre-trained on the CelebFaces Attributes Dataset (CelebA), which contains over 200,000 images of celebrities annotated with 40 binary attributes. For evaluation, the authors generated their own "target" faces using the GAN to ensure a perfect likeness was mathematically possible within the system's search space.

Key Results

The authors conducted a study with 13 "constructors" (people making the faces) and 55 "evaluators" (people judging the results).

  • Recognition Success: In a lineup-style test, 10 out of 16 generated composites were correctly identified by more than 50% of evaluators. Seven of those were recognized by over 75% of users.
  • Efficiency: In one documented session, a user successfully reconstructed a likeness of actor Chris Hemsworth in 46 generations, taking roughly 35 minutes.
  • Novice Performance: Interestingly, the "best" composites (highest similarity scores) were often generated by novice users rather than the researchers, suggesting the interface is highly intuitive.
  • Similarity Scores: The average similarity score across all tests was 47.51%, which the authors note is a significant improvement over previous LVE-based baselines that lacked feature locking.
Target images vs. the final CG-GAN composite of Chris Hemsworth.

Discussion and Conclusion

CG-GAN successfully marries holistic recognition with constructive control. By removing the forensic artist from the loop, the system empowers casual users to translate their memories into photorealistic images directly. The use of orthogonalization to "disentangle" features is a significant UX solution to the problem of latent space entanglement.

However, limitations remain. Since the model was trained on CelebA, it struggles with features not common in celebrity photos, such as scars or unique facial deformities. Future work could explore unsupervised learning to discover more nuanced facial measures and account for environmental factors like lighting or different poses. Despite these challenges, CG-GAN represents a major step forward in "artist-free" forensic technology.

Summary updated

Abstract

Facial composites are graphical representations of an eyewitness's memory of a face. Many digital systems are available for the creation of such composites but are either unable to reproduce features unless previously designed or do not allow holistic changes to the image. In this paper, we improve the efficiency of composite creation by removing the reliance on expert knowledge and letting the system learn to represent faces from examples. The novel approach, Composite Generating GAN (CG-GAN), applies generative and evolutionary computation to allow casual users to easily create facial composites. Specifically, CG-GAN utilizes the generator network of a pg-GAN to create high-resolution human faces. Users are provided with several functions to interactively breed and edit faces. CG-GAN offers a novel way of generating and handling static and animated photo-realistic facial composites, with the possibility of combining multiple representations of the same perpetrator, generated by different eyewitnesses.
BibTeX
@inproceedings{zaltron2020cg,
  author = {Nicola Zaltron and Luisa Zurlo and Sebastian Risi},
  title = {{CG-GAN: An Interactive Evolutionary GAN-based Approach for Facial Composite Generation)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i03.5637},
}
Figure