Abstract
We present "Splice," a novel framework for semantic appearance transfer that maps the visual style of a target image onto the structure of a source image. By leveraging the internal representations of a pre-trained, self-supervised Vision Transformer (DINO-ViT), Splice achieves high-resolution, semantically aware results using only a single pair of images. The method avoids complex adversarial training and manual labels by disentangling appearance and structure through specific ViT feature components.
Introduction
Semantic appearance transfer aims to "paint" the appearance of a target object onto the structure of a source object while maintaining semantic consistency—for example, ensuring that a spotted cow's texture is correctly mapped onto a red cow's body parts despite differences in pose or lighting.
Traditional approaches often struggle with this task. Neural Style Transfer (NST) typically fails to respect object boundaries, often applying textures indiscriminately. Conversely, Generative Adversarial Network (GAN) based methods require massive datasets to learn domain-specific mappings (e.g., horse-to-zebra). Splice addresses these limitations by enabling "in-the-wild" transfer for arbitrary image pairs without requiring large-scale training data or manual segmentation masks.
Methodology: The Splicing Recipe
The core innovation of Splice lies in "untwisting" the internal features of a pre-trained DINO-ViT to isolate style from geometry. The framework utilizes a lightweight U-Net generator ($G_\theta$) trained on the input image pair through internal learning and data augmentation.
Disentangling Appearance and Structure
Splice relies on two distinct representations extracted from the ViT:
- Appearance Representation: The authors utilize the [CLS] token from the deepest layer of the ViT. This token serves as a global summary of appearance that is "spatially flexible," capturing what an object looks like (texture and parts) without being tied to a specific location.
- Structure Representation: To preserve the layout of the source image, the model uses the self-similarity of "keys" in the self-attention layers. By analyzing how patches relate to one another spatially, the model captures shape and structure independently of color or texture.
Objective Function
The generator is optimized using a dual-loss approach. The Appearance Loss ($\mathcal{L}_{app}$) encourages the output to match the [CLS] token of the target image, while the Structure Loss ($\mathcal{L}_{structure}$) ensures the output maintains the key self-similarity of the source image. This allows the model to find meaningful semantic correspondences—such as matching a bird's beak in the target to the beak in the source—purely through self-supervised features.
Key Results and Evaluation
The authors evaluated Splice on several datasets, including AFHQ (Animal Faces), Flickr Mountain, and a new "Wild-Pairs" dataset featuring high-resolution images of diverse objects in complex settings.
- Qualitative Performance: Splice successfully transfers intricate patterns, such as leopard spots onto a dog, while maintaining the original pose. It avoids common pitfalls like color bleeding or structural collapse.
- User Preference: In Amazon Mechanical Turk (AMT) studies, users preferred Splice over state-of-the-art baselines like STROTSS or Swapping Autoencoders by significant margins (often 20-30%).
- Layout Preservation: Quantitative analysis using Mean IoU of semantic masks confirmed that Splice preserves the original scene layout more effectively than most competing methods.
Limitations and Future Work
The framework's performance is intrinsically linked to the pre-trained ViT's knowledge. If the ViT has not encountered a specific object category or cannot find a semantic bridge between two highly unrelated categories (e.g., a bird and an airplane), the transfer may result in hallucinated features or failure. Future research may extend this "splicing" concept to video or 3D media, leveraging self-supervised features for advanced editing tasks.
Summary updated