Son of Zorn's lemma: Targeted style transfer using instance-aware semantic segmentation

Carlos Domingo Castillo, Soham De, Xintong Han, Bharat Singh, Abhay Kumar Yadav, Tom Goldstein

Abstract

Traditional neural style transfer techniques apply artistic filters globally, which is often unsuitable for applications requiring specific object stylization within photorealistic backgrounds. We present a targeted style transfer pipeline that integrates global style transfer, instance-aware semantic segmentation, and Markov Random Field (MRF) based blending. By treating the boundary between stylized and original pixels as an optimization problem, our method mitigates common artifacts such as "halos" and aliasing, producing seamless transitions between stylized objects and real-world environments.

Introduction

Since the landmark work of Gatys et al. in 2015, neural style transfer has become a staple of computer vision, enabling the transformation of any photograph into the style of a master painter like Van Gogh or Picasso. However, these traditional methods are inherently "global," meaning they apply the style to every pixel in the frame uniformly.

This global application poses a significant challenge for modern media applications, such as augmented reality (e.g., Pokémon GO) or hybrid live-action/animation productions (e.g., the TV show Son of Zorn). In these contexts, creators often want a specific character or object to appear stylized while the surrounding environment remains photorealistic. Simply cutting out a stylized object and pasting it onto a photo usually results in jarring artifacts, including "halos," aliasing, and unnatural color transitions at the boundaries. This paper seeks to bridge this gap by performing "targeted" transfer that allows stylized objects to blend naturally into their original surroundings.

Methodology

The proposed pipeline achieves seamless targeted style transfer through a three-stage process:

  1. Global Style Transfer: The system first applies the style of a source template image to the entire target image using the standard Gatys method. This creates a fully stylized version of the scene to serve as the source for the object of interest.
  2. Instance-Aware Semantic Segmentation: To isolate the specific object, the authors utilize a Multi-task Network Cascade (MNC). This deep network identifies specific object instances (e.g., a specific dog or car) and generates a precise binary mask.
  3. MRF-Based Blending: This is the core innovation of the work. Rather than a simple cut-and-paste operation, the boundary is treated as an optimization problem. A Markov Random Field (MRF) decides, pixel-by-pixel, whether to utilize the stylized or original version within a "band of ambiguity" around the object's edge.
The targeted style transfer pipeline: combining global style transfer, instance-aware segmentation, and MRF-based blending.

The MRF optimization relies on two primary components:

  • Unary Potentials: These encourage pixels deep inside the segmentation mask to be stylized and pixels far outside to remain original.
  • Binary Potentials: These ensure the transition between stylized and original pixels occurs where it is least noticeable (e.g., where intensity differences are minimal), effectively providing high-quality anti-aliasing.

Datasets and Implementation

The research leverages existing high-performance architectures trained on standard benchmarks rather than introducing a new dataset. The MNC used for segmentation was pre-trained on the PASCAL VOC and MS COCO datasets. The style transfer component utilizes the VGG-19 network pre-trained on ImageNet. Qualitative evaluation was performed on a diverse set of real-world images featuring common objects like dogs, cars, cats, and birds.

Key Results and Insights

The results demonstrate that the MRF-based approach significantly outperforms "naive" masking. Because style transfer often involves dramatic shifts in color and texture, the "seam" where a stylized object meets a photorealistic background is typically a high-contrast eyesore. The MRF acts as a "smart glue," allowing the stylized object to "bleed" slightly into the background or vice versa to hide the transition.

Qualitative comparison between Simple Transfer (hard masking) and MRF Transfer across various subjects.

Key observations from the results include:

  • Edge Smoothing: The MRF approach eliminated the jagged "staircase" artifacts frequently seen on the roof of stylized cars or the contours of animals.
  • Natural Transitions: In complex scenarios, such as a bird on a branch, the MRF successfully blended fine details like the bird’s tail into the background, whereas simple masking left a visible, unnatural border.
  • Morphological Consistency: The method successfully re-textures objects while maintaining their original shape and structural integrity.

Limitations and Future Work

While effective, the current method is a "hand-off" between three distinct systems (Style Transfer, Segmentation, and MRF) that are not yet trained end-to-end. Additionally, the iterative optimization process required for style transfer remains computationally expensive, limiting real-time applications. Future research should focus on integrating these stages into a single, unified deep neural network capable of performing targeted style transfer in a single forward pass, which would be essential for video or AR integration.

Summary updated

Abstract

Style transfer is an important task in which the style of a source image is mapped onto that of a target image. The method is useful for synthesizing derivative works of a particular artist or specific painting. This work considers targeted style transfer, in which the style of a template image is used to alter only part of a target image. For example, an artist may wish to alter the style of only one particular object in a target image without altering the object's general morphology or surroundings. This is useful, for example, in augmented reality applications (such as the recently released Pokémon go), where one wants to alter the appearance of a single real-world object in an image frame to make it appear as a cartoon. Most notably, the rendering of real-world objects into cartoon characters has been used in a number of films and television show, such as the upcoming series Son of Zorn. We present a method for targeted style transfer that simultaneously segments and stylizes single objects selected by the user. The method uses a Markov random field model to smooth and anti-alias outlier pixels near object boundaries, so that stylized objects naturally blend into their surroundings.
BibTeX
@inproceedings{castillo2017son,
  author = {Carlos Domingo Castillo and Soham De and Xintong Han and Bharat Singh and Abhay Kumar Yadav and Tom Goldstein},
  title = {{Son of Zorn's lemma: Targeted style transfer using instance-aware semantic segmentation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=mOYRd9nJQs},
}
Figure