Abstract
Facial expression editing often struggles with the simultaneous control of head pose and the preservation of facial identity. POCE (Pose-Controllable Expression editing) introduces a framework that enables joint manipulation of expression and pose using only unpaired training images. By performing editing in a pose-invariant 2D UV space, the model achieves high-fidelity results and superior identity preservation, bypassing the need for difficult-to-collect paired datasets.
Introduction
The task of facial expression editing faces a persistent "trilemma": entanglement, data scarcity, and visual fidelity. In real-world scenarios, expressions and head poses are inherently linked; however, most models fail to decouple them, often leading to identity distortion when the pose is altered. Furthermore, high-quality pose control typically requires paired data—images of the same individual in various poses—which is challenging to acquire at scale. Finally, standard encoder-decoder networks frequently produce blurry results in highly mobile regions like the mouth and eyes.
POCE addresses these challenges by decoupling facial texture from geometry. By leveraging 3D modeling and UV mapping, the framework provides a robust solution for high-fidelity, pose-controllable expression editing.
Methodology
The POCE pipeline is structured into four distinct stages designed to isolate and manipulate facial attributes effectively.
1. 3D Face Fitting and UV Sampling
The process begins by using a pre-trained 3D Morphable Model (3DDFA) to estimate the 3D shape and pose from a 2D source image. The facial texture is then "unwrapped" into a UV map. This 2D template ensures that facial features—such as the eyes, nose, and mouth—are consistently located at the same pixel coordinates, effectively neutralizing the original head pose.
2. Self-Supervised UV Completion
Because a single 2D image cannot capture the entire head, the initial UV map contains "holes" caused by self-occlusion. POCE employs a patch-based GAN to fill these gaps. This component is self-supervised; it learns the distribution of valid facial patches across a broad dataset of incomplete faces, eliminating the need for expensive 3D scans or ground-truth complete UV maps.
3. Modulation-based UV Editing
With a complete UV map, the model performs expression editing using Relative Action Units (AUs). Rather than providing absolute commands, the model is instructed to adjust AU values relative to the source (e.g., "increase the smile value by X"). This residual learning approach allows the network to focus exclusively on regions requiring change, which significantly enhances identity preservation. The architecture utilizes StyleGAN-like modulation to maintain high visual sharpness.
4. Rendering
The final stage involves wrapping the edited UV map back onto the 3D geometry. The face can then be rendered at any target pose requested by the user, providing seamless control over both expression and orientation.
Experimental Results
POCE was evaluated against state-of-the-art (SOTA) methods, including GANimation, MSF, and FOMM, across three major benchmarks: MultiPIE, RaFD, and 300W-LP.
Quantitative Performance
The model demonstrated superior performance in both identity retention and image quality:
- Identity Preservation: POCE achieved the lowest Identity Embedding Distance (IED), confirming that the edited images more closely resemble the original subject compared to competing models.
- Visual Quality: On the RaFD dataset, POCE reached a Fréchet Inception Distance (FID) of 7.41, a significant improvement over GANimation’s 14.18.
Qualitative Assessment
In user studies focusing on realism, participants consistently rated POCE-generated images as more realistic and more faithful to the reference expressions than those produced by SOTA methods. The model also demonstrated a unique capability for "expression transplanting," where expressions are copied from a reference UV map to a source UV map via binary masking, yielding high-quality results without manual annotation.
Discussion and Limitations
A key insight of POCE is the use of UV space as a "disentangler." By shifting editing to a pose-invariant space, the neural network can focus purely on texture (expression) while the geometry handles pose. This prevents the common issue of face-shape warping during head rotation.
However, some limitations remain. The UV maps primarily cover facial skin, necessitating the use of Poisson blending to reintegrate the edited face into the original photo. This can occasionally result in artifacts if lighting or hair overlaps the face significantly. Additionally, extremely sharp profile poses remain a challenge for UV completion due to the limited visible data.
Conclusion
POCE provides a powerful framework for pose-controllable expression editing that circumvents the need for paired training data. By operating in UV space and utilizing relative AU modulation, it sets a new standard for identity preservation and visual fidelity. As with all generative technologies, the authors acknowledge the potential for misuse in creating synthetic media and advocate for responsible deployment strategies, such as watermarking.
Summary updated