Abstract
Domain adaptation for Generative Adversarial Networks (GANs) typically focuses on transferring artistic styles from a target domain to a pre-trained source model. However, real-world shifts often involve new physical "entities"—such as masks or hats—that are absent in the source data. This paper presents a framework for generalized one-shot GAN adaptation that simultaneously transfers style and incorporates novel entities from a single reference image. By decoupling style from entity generation and employing variational Laplacian regularization, the proposed method maintains high image quality and cross-domain correspondence, significantly outperforming existing one-shot adaptation techniques.
Introduction
Pre-trained GANs like StyleGAN are powerful generative tools but require extensive datasets for training. Domain adaptation offers a path to repurpose these models for new domains with minimal data. The most challenging scenario is "one-shot" adaptation, where only a single target image is available. While previous methods have successfully addressed style transfer (texture and color), they often struggle with "entities"—structural additions like glasses or headwear. These methods frequently produce artifacts or fail to preserve the identity of the source subject when attempting to incorporate new objects.
The authors address this "generalized" adaptation task: the simultaneous transfer of global style and local entities while preserving the underlying content and identity of the source domain.
Methodology
The proposed framework utilizes a three-pronged strategy to ensure stable and high-quality adaptation:
- Decoupled Architecture: To prevent style-entity interference, the system uses an auxiliary UNet alongside the main generator. The primary generator handles global stylization, while the auxiliary network focuses on local entity generation, guided by a binary mask provided with the target image.
- Internal Distribution Learning: Traditional adversarial training is unstable with a single image. Instead, the authors employ Sliced Wasserstein Distance (SWD) to match the patch-level statistical distributions between the target image and the generated outputs, ensuring more robust one-shot learning.
- Variational Laplacian Regularization: To maintain "cross-domain correspondence"—ensuring the generated subject retains the identity of the source subject—the authors apply Laplacian regularization. This constrains the transformation's smoothness, preserving the geometric manifold of the original GAN.
Results and Evaluation
The method was evaluated using source domains like FFHQ (faces) and LSUN Church against diverse target domains including artistic styles (Sketch, Disney, Arcane) and structural entities (masks, Zelda decorations, hats).
Quantitative and Qualitative Performance
The framework demonstrated a clear edge over baselines like JoJoGAN and OSCLIP:
- Identity Preservation: The model achieved superior scores in Normalized Mean Error (NME) for facial landmarks and Identity Similarity (IS), confirming its ability to maintain the subject's original features.
- User Preference: In a study involving 53 volunteers, the proposed method was preferred in approximately 60% of cases for both quality and correspondence.
- Efficiency: Training is highly efficient, requiring only 3 to 12 minutes on a single NVIDIA RTX 3090.
Discussion and Limitations
The core innovation lies in treating style and entities as separate but complementary tasks. By moving away from purely adversarial training toward distribution matching and manifold preservation, the authors achieve a level of stability previously unseen in one-shot adaptation.
However, limitations remain. The model is sensitive to pose; if the generated face's orientation differs significantly from the reference image, entity placement (like a hat) may be imprecise. Additionally, extremely complex structural changes remain difficult to capture perfectly from a single example.
Conclusion
This work advances one-shot GAN adaptation by enabling the integration of new physical entities alongside artistic styles. The combination of a decoupled architecture and Laplacian regularization provides a robust solution for creative applications where data is scarce but structural fidelity is paramount.
Summary updated