Abstract
Identity Stylization (IDS) is a novel task that aims to adapt a face to a new domain—such as a 3D cartoon, sketch, or oil painting—while strictly preserving the original person's unique facial structure and identity. We present Styleverse, a framework that leverages a single domain-aware generator and a novel 13-domain dataset (FS13) to achieve high-fidelity, topology-aware results. By focusing on facial geometry rather than just texture, Styleverse outperforms traditional style transfer and face-swapping methods across heterogeneous domains.
Introduction
Current image synthesis tasks typically fall into two categories: Style Transfer and Identity Swapping. Style transfer methods (e.g., AdaIN, WCT) excel at texture migration but often distort facial "topology"—the specific geometric arrangement of features that makes a person recognizable. Conversely, identity swapping (Deepfakes) replaces the source identity entirely.
Styleverse addresses the "missing middle": Identity Stylization (IDS). The goal is to keep the source person's identity while "re-rendering" them into a different domain, creating what the authors call a "Parallel Style Universe." This requires a model capable of handling heterogeneous domains (e.g., infrared images vs. Beijing Opera makeup) using a single, efficient architecture.
The Styleverse Framework
The Styleverse framework is designed for many-to-many domain translation using a universal generator rather than separate models for every domain pair.
Key Components:
- Heterogeneous Encoders: A Texture Encoder (VGG19) extracts content features, while a Distinctive Style (DS) Encoder (LightCNN) extracts domain-aware embeddings. LightCNN was found to be superior to standard identity networks like ArcFace because its attention is more tightly coupled to the topological facial structures necessary for stylization.
- Universal Style Modulation (USM): This module integrates a "DS code" (global domain style) and a "diversity code" (instance-specific texture) to control the generator via multi-scale Adaptive Instance Normalization (AdaIN).
- Loss Functions: The model is trained using a combination of Reconstruction loss, a Heterogeneous domain-aware prior consistency loss (to ensure target domain fidelity), and Contextual losses to maintain spatial structure.
The FS13 Dataset
To evaluate IDS, the authors introduced FS13 (Face Styles 13), a dataset containing 13 heterogeneous domains ranging from real-world daytime/night portraits and NIR images to artistic styles like 3D cartoons, anime, sculptures, and Beijing Opera makeup. FS13 is a "few-shot" challenge, providing only 100 samples per domain, which forces the model to learn generalized domain features rather than memorizing specific images.
Results and Discussion
Styleverse achieves state-of-the-art performance in both quantitative and qualitative metrics. Quantitatively, it achieved the best $ID_{style}$ scores across nearly all categories in the FS13 benchmark. Qualitatively, visual comparisons demonstrate that Styleverse preserves eye shape and facial contours significantly better than methods like AdaIN or SANet, which often produce distorted features when moving between disparate domains.
A key insight from the research is that identity in stylization is fundamentally about preserving underlying facial geometry (topology). The researchers also identified a "PSU Collapse" failure mode where models struggle to balance style intensity with content preservation over many training iterations.
Conclusion
Styleverse provides a robust solution for identity-preserving stylization across diverse domains. While the model currently faces challenges with background distortion and complex occlusions (like glasses), it offers a significant leap in computational efficiency and topological fidelity. Future work may extend this concept to 3D face stylization and non-human object domains.
Summary updated