StyleYourSmile: Cross-Domain Face Retargeting Without Paired Multi-Style Data

Avirup Dey, Vinay NamboodiriarXiv 2025 · DOI · arXiv

Abstract

Face retargeting across artistic domains—such as transferring a smile from a photograph to a classical oil painting—presents a significant challenge in computer vision due to the need to disentangle identity, expression, and domain-specific style. StyleYourSmile is a novel one-shot framework that achieves high-fidelity cross-domain retargeting without requiring expensive, curated multi-style paired datasets. By leveraging a dual-encoder architecture and a training-free data augmentation strategy, the model preserves identity and artistic nuances while accurately following driving expressions.

Introduction

Face retargeting involves transferring the facial expressions and poses of a "driving" image to a "source" identity. While well-studied in the photo-realistic domain, cross-domain retargeting remains difficult. The primary hurdle is the effective disentanglement of three distinct signals: identity (unique facial features), expression (transient movements), and style (domain-specific attributes like brushstrokes or lighting). Existing methods often rely on rare paired data or discard stylistic details in favor of identity recognition, leading to suboptimal results in artistic contexts.

Methodology

The StyleYourSmile framework utilizes a dual-encoder architecture conditioned on Stable Diffusion v1.5, enhanced by efficient fine-tuning and novel data augmentation.

Dual-Encoder Architecture

To address the "stripped-down" representation problem where face recognition models discard stylistic nuances, the authors employ two separate encoders:

  1. Identity Encoder ($\mathcal{E}_{id}$): An ArcFace model extracts domain-invariant identity features, projected as "identity tokens" into the CLIP text space.
  2. Style Encoder ($\mathcal{E}_{sty}$): A CLIP visual encoder captures perceptual details like lighting and texture, projected as "style tokens."

Spatial and Expression Control

Expression transfer is guided by 3D Morphable Model (3DMM) landmarks and foreground masks. These spatial priors are processed through a ControlNet module. Interestingly, the authors found that routing style tokens through the ControlNet branch, rather than directly into the UNet's cross-attention layers, better preserved fine-grained identity features.

3DMM landmarks used for spatial control and expression transfer.

StyleInject Data Augmentation

To overcome the lack of paired multi-style data, the authors introduced StyleInject, a training-free style transfer technique. By augmenting standard video datasets (e.g., VoxCeleb1) with synthetic styles, the model learns to decouple identity from style by observing the same content across various visual domains during training. The UNet is fine-tuned using Low-Rank Adaptation (LoRA) to maintain the generative priors of the base model.

Experimental Results

StyleYourSmile was evaluated against state-of-the-art baselines, including HyperReenact, DiffusionRig, and Arc2Face, using the VoxCeleb1 and GeneFace datasets.

Quantitative Performance

The model demonstrated superior reconstruction quality and identity preservation. On the stylized VoxCeleb1 test set, it achieved a PSNR of 19.889 and an LPIPS of 0.146, significantly outperforming DiffusionRig (PSNR 13.650) and HyperReenact (LPIPS 0.377). It also maintained a high Cosine Similarity for Identity (CS-ID) of 0.615.

Qualitative Analysis

Qualitatively, StyleYourSmile excels at preserving domain-specific attributes—such as the grain of vintage photos or the texture of oil paintings—while accurately mirroring the driving expression.

Cross-domain face retargeting results across various artistic styles and identities.

Conclusion and Future Work

StyleYourSmile provides a robust solution for one-shot cross-domain face retargeting. By avoiding the need for paired data and utilizing a dual-encoder approach, it achieves high-fidelity results across diverse artistic styles. Future work will address current limitations, such as performance on highly abstract domains (e.g., anime) where face detection fails, and improving computational efficiency for real-time applications.

Summary updated

BibTeX
@misc{dey2025styleyoursmile,
  author = {Avirup Dey and Vinay Namboodiri},
  title = {{StyleYourSmile: Cross-Domain Face Retargeting Without Paired Multi-Style Data}},
  year = {2025},
  eprint = {2512.01895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.01895},
}
Figure