Styleverse: Towards Identity Stylization across Heterogeneous Domains

Jia Li, Jie Cao, Junxian Duan, Ran HearXiv 2022 · DOI · arXiv

Abstract

Identity Stylization (IDS) is a novel task that aims to adapt a face to a new domain—such as a 3D cartoon, sketch, or oil painting—while strictly preserving the original person's unique facial structure and identity. We present Styleverse, a framework that leverages a single domain-aware generator and a novel 13-domain dataset (FS13) to achieve high-fidelity, topology-aware results. By focusing on facial geometry rather than just texture, Styleverse outperforms traditional style transfer and face-swapping methods across heterogeneous domains.

Introduction

Current image synthesis tasks typically fall into two categories: Style Transfer and Identity Swapping. Style transfer methods (e.g., AdaIN, WCT) excel at texture migration but often distort facial "topology"—the specific geometric arrangement of features that makes a person recognizable. Conversely, identity swapping (Deepfakes) replaces the source identity entirely.

Styleverse addresses the "missing middle": Identity Stylization (IDS). The goal is to keep the source person's identity while "re-rendering" them into a different domain, creating what the authors call a "Parallel Style Universe." This requires a model capable of handling heterogeneous domains (e.g., infrared images vs. Beijing Opera makeup) using a single, efficient architecture.

The Styleverse Framework

The Styleverse framework is designed for many-to-many domain translation using a universal generator rather than separate models for every domain pair.

The Styleverse architecture featuring heterogeneous encoders and Universal Style Modulation (USM).

Key Components:

  • Heterogeneous Encoders: A Texture Encoder (VGG19) extracts content features, while a Distinctive Style (DS) Encoder (LightCNN) extracts domain-aware embeddings. LightCNN was found to be superior to standard identity networks like ArcFace because its attention is more tightly coupled to the topological facial structures necessary for stylization.
  • Universal Style Modulation (USM): This module integrates a "DS code" (global domain style) and a "diversity code" (instance-specific texture) to control the generator via multi-scale Adaptive Instance Normalization (AdaIN).
  • Loss Functions: The model is trained using a combination of Reconstruction loss, a Heterogeneous domain-aware prior consistency loss (to ensure target domain fidelity), and Contextual losses to maintain spatial structure.

The FS13 Dataset

To evaluate IDS, the authors introduced FS13 (Face Styles 13), a dataset containing 13 heterogeneous domains ranging from real-world daytime/night portraits and NIR images to artistic styles like 3D cartoons, anime, sculptures, and Beijing Opera makeup. FS13 is a "few-shot" challenge, providing only 100 samples per domain, which forces the model to learn generalized domain features rather than memorizing specific images.

Results and Discussion

Styleverse achieves state-of-the-art performance in both quantitative and qualitative metrics. Quantitatively, it achieved the best $ID_{style}$ scores across nearly all categories in the FS13 benchmark. Qualitatively, visual comparisons demonstrate that Styleverse preserves eye shape and facial contours significantly better than methods like AdaIN or SANet, which often produce distorted features when moving between disparate domains.

Qualitative comparison of Styleverse against SOTA style transfer methods across various domains.

A key insight from the research is that identity in stylization is fundamentally about preserving underlying facial geometry (topology). The researchers also identified a "PSU Collapse" failure mode where models struggle to balance style intensity with content preservation over many training iterations.

Conclusion

Styleverse provides a robust solution for identity-preserving stylization across diverse domains. While the model currently faces challenges with background distortion and complex occlusions (like glasses), it offers a significant leap in computational efficiency and topological fidelity. Future work may extend this concept to 3D face stylization and non-human object domains.

Summary updated

Abstract

We propose a new challenging task namely IDentity Stylization (IDS) across heterogeneous domains. IDS focuses on stylizing the content identity, rather than completely swapping it using the reference identity. We use an effective heterogeneous-network-based framework $Styleverse$ that uses a single domain-aware generator to exploit the Metaverse of diverse heterogeneous faces, based on the proposed dataset FS13 with limited data. FS13 means 13 kinds of Face Styles considering diverse lighting conditions, art representations and life dimensions. Previous similar tasks, \eg, image style transfer can handle textural style transfer based on a reference image. This task usually ignores the high structure-aware facial area and high-fidelity preservation of the content. However, Styleverse intends to controllably create topology-aware faces in the Parallel Style Universe, where the source facial identity is adaptively styled via AdaIN guided by the domain-aware and reference-aware style embeddings from heterogeneous pretrained models. We first establish the IDS quantitative benchmark as well as the qualitative Styleverse matrix. Extensive experiments demonstrate that Styleverse achieves higher-fidelity identity stylization compared with other state-of-the-art methods.
BibTeX
@misc{li2022styleverse,
  author = {Jia Li and Jie Cao and Junxian Duan and Ran He},
  title = {{Styleverse: Towards Identity Stylization across Heterogeneous Domains}},
  year = {2022},
  eprint = {2203.00861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.00861},
}
Figure