Modify: Model-Driven Face Stylization Without Style Images

Yuhe Ding, Jian Liang, Jie Cao, Aihua Zheng, Ran He

Introduction

Traditional unsupervised face stylization frameworks, such as CycleGAN or StarGAN, typically rely on a data-driven paradigm. These methods require simultaneous access to both source (content) and target (style) datasets during the training process. However, this requirement presents significant challenges in real-world applications. Privacy regulations often prevent the sharing of sensitive user photos, and high-quality artistic datasets are frequently protected by restrictive copyright licenses. Furthermore, these models often struggle with generalization when faced with distribution shifts in test data.

To address these hurdles, we present MODIFY (MODel-drIven Face stYlization), a framework that shifts the focus from data-driven to model-driven translation. By encapsulating style information within a pre-trained generative model, MODIFY enables face stylization without requiring access to the original target style images during the translation phase, thereby protecting data privacy and reducing logistical costs.

The MODIFY Framework

The MODIFY framework operates in two distinct stages: Style Encapsulation and Face Stylization.

1. Style Encapsulation

In the first stage, the model learns the target style using only the target dataset. The architecture integrates three primary components:

  • FPN Encoder (E): Maps images into a latent space.
  • StyleGAN Decoder (D): Renders latent codes into high-quality images.
  • Remapping Network (M): Maps a known distribution (e.g., Gaussian noise) into the higher-resolution portions of the latent space to capture multimodal style information.

The core innovation of this stage is the Swapping Loss. This loss function ensures that the encoder and the remapping network share a common embedding space. By swapping latent codes and enforcing reconstruction, the model learns to effectively disentangle content from style, "baking" the artistic characteristics into the decoder.

2. Face Stylization

Once the style is encapsulated, the target images are no longer needed and can be discarded. When a source image is provided for stylization, the system fine-tunes a specific encoder ($E'$) to map the source content into the pre-trained style space. During this process, the decoder remains frozen to preserve the learned style characteristics.

Versatile Training Paradigms

A major strength of the MODIFY framework is its flexibility. It supports three distinct training modes:

  • Offline Training: Standard batch processing for large-scale translation.
  • Online Training: Updating the model incrementally as images arrive.
  • Test-time Training: Adaptively fine-tuning the encoder for a specific input image in as few as 50 iterations. This capability makes the system highly robust to distribution shifts, such as unusual lighting or poses, which often cause static models to fail.

Experimental Evaluation

Datasets

The authors evaluated MODIFY using two primary datasets: MetFaces (1,336 high-quality artistic face images) as the target style domain, and FFHQ (2,000 high-quality human face photos) as the source content domain.

Quantitative and Qualitative Results

MODIFY was compared against state-of-the-art methods, including StarGAN v2 and Pixel2Style2Pixel (PSP).

  • FID Scores: MODIFY achieved a Frechet Inception Distance (FID) of 58.6, significantly outperforming StarGAN v2 (65.5) and an unpaired version of PSP (74.2). A lower FID indicates higher image quality and better style alignment.
  • User Study: In a voting study involving 100 subjects, MODIFY received a 45.2% approval rate. This is nearly identical to the 45.9% achieved by the data-driven "Unpaired PSP," suggesting that MODIFY achieves comparable quality to data-heavy methods while maintaining strict privacy.
  • Ablation Studies: Results confirmed that the Swapping Loss is essential for success; without it, generated images exhibit significant artifacts and poor style encapsulation.

Discussion and Limitations

While MODIFY offers a powerful privacy-preserving paradigm, it involves a trade-off controlled by a hyper-parameter ($\xi$). As the model incorporates more style layers (increasing $\xi$), the artistic effect becomes more pronounced, but the original identity of the subject may begin to fade. Balancing identity preservation with stylistic transformation remains a key consideration for users.

Conclusion

MODIFY represents a significant step forward in face stylization by decoupling the style acquisition phase from the translation phase. By moving to a model-driven approach, researchers and developers can share pre-trained style models without compromising raw data privacy. Future work may extend this concept to other domains where privacy is critical, such as medical imaging or satellite data translation, and explore improved methods for identity maintenance through facial landmark integration.

Summary updated

BibTeX
@inproceedings{ding2023modify,
  author = {Yuhe Ding and Jian Liang and Jie Cao and Aihua Zheng and Ran He},
  title = {{Modify: Model-Driven Face Stylization Without Style Images}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095222},
}
Figure