Abstract
Face retargeting across artistic domains—such as transferring a smile from a photograph to a classical oil painting—presents a significant challenge in computer vision due to the need to disentangle identity, expression, and domain-specific style. StyleYourSmile is a novel one-shot framework that achieves high-fidelity cross-domain retargeting without requiring expensive, curated multi-style paired datasets. By leveraging a dual-encoder architecture and a training-free data augmentation strategy, the model preserves identity and artistic nuances while accurately following driving expressions.
Introduction
Face retargeting involves transferring the facial expressions and poses of a "driving" image to a "source" identity. While well-studied in the photo-realistic domain, cross-domain retargeting remains difficult. The primary hurdle is the effective disentanglement of three distinct signals: identity (unique facial features), expression (transient movements), and style (domain-specific attributes like brushstrokes or lighting). Existing methods often rely on rare paired data or discard stylistic details in favor of identity recognition, leading to suboptimal results in artistic contexts.
Methodology
The StyleYourSmile framework utilizes a dual-encoder architecture conditioned on Stable Diffusion v1.5, enhanced by efficient fine-tuning and novel data augmentation.
Dual-Encoder Architecture
To address the "stripped-down" representation problem where face recognition models discard stylistic nuances, the authors employ two separate encoders:
- Identity Encoder ($\mathcal{E}_{id}$): An ArcFace model extracts domain-invariant identity features, projected as "identity tokens" into the CLIP text space.
- Style Encoder ($\mathcal{E}_{sty}$): A CLIP visual encoder captures perceptual details like lighting and texture, projected as "style tokens."
Spatial and Expression Control
Expression transfer is guided by 3D Morphable Model (3DMM) landmarks and foreground masks. These spatial priors are processed through a ControlNet module. Interestingly, the authors found that routing style tokens through the ControlNet branch, rather than directly into the UNet's cross-attention layers, better preserved fine-grained identity features.
StyleInject Data Augmentation
To overcome the lack of paired multi-style data, the authors introduced StyleInject, a training-free style transfer technique. By augmenting standard video datasets (e.g., VoxCeleb1) with synthetic styles, the model learns to decouple identity from style by observing the same content across various visual domains during training. The UNet is fine-tuned using Low-Rank Adaptation (LoRA) to maintain the generative priors of the base model.
Experimental Results
StyleYourSmile was evaluated against state-of-the-art baselines, including HyperReenact, DiffusionRig, and Arc2Face, using the VoxCeleb1 and GeneFace datasets.
Quantitative Performance
The model demonstrated superior reconstruction quality and identity preservation. On the stylized VoxCeleb1 test set, it achieved a PSNR of 19.889 and an LPIPS of 0.146, significantly outperforming DiffusionRig (PSNR 13.650) and HyperReenact (LPIPS 0.377). It also maintained a high Cosine Similarity for Identity (CS-ID) of 0.615.
Qualitative Analysis
Qualitatively, StyleYourSmile excels at preserving domain-specific attributes—such as the grain of vintage photos or the texture of oil paintings—while accurately mirroring the driving expression.
Conclusion and Future Work
StyleYourSmile provides a robust solution for one-shot cross-domain face retargeting. By avoiding the need for paired data and utilizing a dual-encoder approach, it achieves high-fidelity results across diverse artistic styles. Future work will address current limitations, such as performance on highly abstract domains (e.g., anime) where face detection fails, and improving computational efficiency for real-time applications.
Summary updated