Cross-Species 3D Face Morphing via Alignment-Aware Controller

Xirui Yan, Zhenbo Yu, Bingbing Ni, Hang WangAAAI 2022 · DOI

Introduction

Cross-species 3D face morphing—transforming a detailed human face into an animal face while preserving the individual's unique identity—presents a significant challenge in computer graphics. Unlike standard human-to-human morphing, cross-species tasks must navigate a massive geometric shift (e.g., from a human nose to a cat snout) without losing fine-grained details like wrinkles, beards, or eye shapes. Existing methods often result in "mushy" geometries that fail to capture either the target species' characteristics or the source's identity.

Overview of the AFM framework morphing a human face into a cat while preserving identity features.

The Alignment-aware 3D Face Morphing (AFM) Framework

To address these challenges, the researchers developed the Alignment-aware 3D Face Morphing (AFM) framework. The system is designed to balance shape alignment with feature preservation through three core innovations.

1. Explicit Controller (EC)

Rather than relying on implicit neural representations, AFM uses an "Explicit Controller"—a simplified 3D cage that encapsulates the face mesh. This controller is trained with specific constraints to ensure it aligns semantically with both the source human and the target animal. By deforming this lower-resolution cage, the system gains precise control over the global shape transformation.

2. Controller-Based Mapping (CBM)

To resolve semantic mismatches between species (e.g., the differing relative positions of ears), AFM employs a landmark-based mapping system. It utilizes a "multi-density" correspondence, placing higher importance on critical identity regions like the eyes and nose. This ensures that the most recognizable human features are mapped correctly onto the corresponding regions of the animal geometry.

3. Sparse-to-Dense Inference

Handling high-resolution 3D scans with tens of thousands of vertices is computationally expensive. AFM solves this by deforming a low-resolution version of the face and then applying Mean Value Coordinates (MVC). This mathematical interpolation transfers the complex deformations back to the original high-resolution, textured mesh, preserving intricate skin details.

The AFM architecture showing the interaction between source/target encoders, generators, and the deformation network with associated loss functions.

Training and Evaluation

The model is trained using a suite of loss functions, including 3D Chamfer distance for shape matching and 2D rendering losses for silhouette accuracy. A specialized "Eye-keeping loss" ($\mathcal{L}_{eye}$) is used to prevent distortion in the ocular region, which is vital for human recognition.

The framework was evaluated using the COMA and Headspace datasets, alongside high-quality animal meshes. Quantitatively, AFM outperformed baselines in both Shape Morphing Error (SME) and Feature Preserving Error (FPE). In user studies, it achieved high scores for both morphing quality (8.4/10) and human recognition (8.1/10), demonstrating its ability to maintain identity even through extreme geometric warping.

Conclusion

The AFM framework provides a robust solution for cross-species morphing by combining explicit geometric control with semantic-aware mapping. While currently dependent on predefined landmarks and similar mesh topologies, it sets a new standard for preserving human identity in non-human target domains, with potential applications in digital entertainment and avatar creation.

Summary updated

BibTeX
@inproceedings{yan2022cross,
  author = {Xirui Yan and Zhenbo Yu and Bingbing Ni and Hang Wang},
  title = {{Cross-Species 3D Face Morphing via Alignment-Aware Controller}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i3.20208},
}
Figure