Splicing ViT Features for Semantic Appearance Transfer

Narek Tumanyan, Omer Bar-Tal, Shai Bagon, Tali Dekel

Abstract

We present "Splice," a novel framework for semantic appearance transfer that maps the visual style of a target image onto the structure of a source image. By leveraging the internal representations of a pre-trained, self-supervised Vision Transformer (DINO-ViT), Splice achieves high-resolution, semantically aware results using only a single pair of images. The method avoids complex adversarial training and manual labels by disentangling appearance and structure through specific ViT feature components.

Introduction

Semantic appearance transfer aims to "paint" the appearance of a target object onto the structure of a source object while maintaining semantic consistency—for example, ensuring that a spotted cow's texture is correctly mapped onto a red cow's body parts despite differences in pose or lighting.

Figure 1: Examples of semantic appearance transfer using the Splice framework across various domains including animals, food, and fruit.

Traditional approaches often struggle with this task. Neural Style Transfer (NST) typically fails to respect object boundaries, often applying textures indiscriminately. Conversely, Generative Adversarial Network (GAN) based methods require massive datasets to learn domain-specific mappings (e.g., horse-to-zebra). Splice addresses these limitations by enabling "in-the-wild" transfer for arbitrary image pairs without requiring large-scale training data or manual segmentation masks.

Methodology: The Splicing Recipe

The core innovation of Splice lies in "untwisting" the internal features of a pre-trained DINO-ViT to isolate style from geometry. The framework utilizes a lightweight U-Net generator ($G_\theta$) trained on the input image pair through internal learning and data augmentation.

Disentangling Appearance and Structure

Splice relies on two distinct representations extracted from the ViT:

  1. Appearance Representation: The authors utilize the [CLS] token from the deepest layer of the ViT. This token serves as a global summary of appearance that is "spatially flexible," capturing what an object looks like (texture and parts) without being tied to a specific location.
  2. Structure Representation: To preserve the layout of the source image, the model uses the self-similarity of "keys" in the self-attention layers. By analyzing how patches relate to one another spatially, the model captures shape and structure independently of color or texture.
Figure 2: The Splice architecture utilizing pre-trained DINO-ViT features to calculate appearance and structure losses for the generator.

Objective Function

The generator is optimized using a dual-loss approach. The Appearance Loss ($\mathcal{L}_{app}$) encourages the output to match the [CLS] token of the target image, while the Structure Loss ($\mathcal{L}_{structure}$) ensures the output maintains the key self-similarity of the source image. This allows the model to find meaningful semantic correspondences—such as matching a bird's beak in the target to the beak in the source—purely through self-supervised features.

Key Results and Evaluation

The authors evaluated Splice on several datasets, including AFHQ (Animal Faces), Flickr Mountain, and a new "Wild-Pairs" dataset featuring high-resolution images of diverse objects in complex settings.

  • Qualitative Performance: Splice successfully transfers intricate patterns, such as leopard spots onto a dog, while maintaining the original pose. It avoids common pitfalls like color bleeding or structural collapse.
  • User Preference: In Amazon Mechanical Turk (AMT) studies, users preferred Splice over state-of-the-art baselines like STROTSS or Swapping Autoencoders by significant margins (often 20-30%).
  • Layout Preservation: Quantitative analysis using Mean IoU of semantic masks confirmed that Splice preserves the original scene layout more effectively than most competing methods.

Limitations and Future Work

The framework's performance is intrinsically linked to the pre-trained ViT's knowledge. If the ViT has not encountered a specific object category or cannot find a semantic bridge between two highly unrelated categories (e.g., a bird and an airplane), the transfer may result in hallucinated features or failure. Future research may extend this "splicing" concept to video or 3D media, leveraging self-supervised features for advanced editing tasks.

Summary updated

Abstract

We present a method for semantically transferring the visual appearance of one natural image to another. Specifically, our goal is to generate an image in which objects in a source structure image are “painted” with the visual appearance of their semantically related objects in a target appearance image. Our method works by training a generator given only a single structure/appearance image pair as input. To integrate semantic information into our framework—a pivotal component in tackling this task-our key idea is to leverage a pre-trained and fixed Vision Transformer (ViT) model which serves as an external semantic prior. Specifically, we derive novel representations of structure and appearance extracted from deep ViT features, untwisting them from the learned self-attention modules. We then establish an objective function that splices the desired structure and appearance representations, interweaving them together in the space of ViT features. Our framework, which we term “Splice”, does not involve adversarial training, nor does it require any additional input information such as semantic segmentation or correspondences, and can generate high resolution results, e.g., work in HD. We demonstrate high quality results on a variety of in-the-wild image pairs, under significant variations in the number of objects, their pose and appearance.
BibTeX
@misc{tumanyan2022splicing,
  author = {Narek Tumanyan and Omer Bar-Tal and Shai Bagon and Tali Dekel},
  title = {{Splicing ViT Features for Semantic Appearance Transfer}},
  year = {2022},
  eprint = {2201.00424},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.00424},
}
Figure