RAST: Restorable Arbitrary Style Transfer via Multi-Restoration

Yingnan Ma, Chenqiu Zhao, Xudong Li, Anup BasuWACV 2023

TL;DR

This paper introduces RAST, a novel framework for arbitrary style transfer that tackles the persistent "content leak" problem by reframing it as an image restoration task. By employing a multi-restoration process and introducing new loss functions, RAST achieves superior content preservation and style embedding compared to existing methods.

What Problem Does This Paper Address?

Arbitrary style transfer aims to imbue a content image with the artistic or photographic style of a reference image. While significant progress has been made, a key challenge remains: the "content leak" issue. This occurs when the process of embedding stylistic features inadvertently alters or corrupts the original content of the image. Existing methods often struggle to strike a balance between faithfully transferring style and preserving the integrity of the content. For artistic styles, this can lead to distorted content, while for photo-realistic styles, the focus on content preservation might result in a lack of stylistic detail. This paper addresses this fundamental trade-off by proposing a new perspective: image restoration.

Methods

The core of the RAST framework is its Restorable Arbitrary Style Transfer (RAST) architecture, which utilizes a quadruple-cycle framework for iterative learning. Instead of directly minimizing content and style differences between input and output, RAST focuses on minimizing the difference between restored images and input images, while simultaneously maximizing the style difference between stylized images and input images. This approach aims to disentangle content and style information more effectively.

Key novelties include:

  • Multi-Restoration Process: The framework performs multiple restoration steps. This iterative process allows for the transmission of both content and style information, with the goal of ensuring content preservation and style consistency.
  • Novel Loss Functions:
    • Multi-restoration Loss (Lmulti): This loss function, built upon perceptual loss, calculates feature differences between restored and input images, as well as among the various multi-restored images. It aims to capture errors arising from different transmission processes.
  • Style Difference Loss (Ldiff): This loss function is designed to maximize the style feature difference between content images and stylized images, preventing the stylized images from converging too closely to the content.
  • Network Architecture: The RAST framework employs a transfer block based on SANet [37] for semantic mapping of content and style features via style attention. A pre-trained VGG-19 network serves as the encoder, and a symmetric VGG-19 network acts as the decoder. Two external multi-scale discriminators are also used for realistic-to-artistic and artistic-to-realistic style transfer.

Datasets

  • Training: MS-COCO [32] was used as the content dataset, and WikiArt [24] was used as the style dataset.
  • Evaluation: The testing set consisted of 10000 content images and 10000 style images from MS-COCO and WikiArt, respectively.

Key Insights and Ideas

  • Reframing Style Transfer as Restoration: The most significant insight is the shift in perspective from direct style transfer to an image restoration problem. This allows for a more robust handling of content preservation by focusing on recovering the original content from a stylized version.
  • Iterative Restoration for Disentanglement: The multi-restoration process is crucial. By iteratively restoring images, the framework implicitly disentangles content and style, allowing for more accurate transmission of each.
  • Balancing Content and Style: The proposed loss functions are key to achieving this balance. Lmulti ensures content and style are preserved through restoration, while Ldiff actively pushes for stylistic distinctiveness.
  • Quantitative Evaluation: The paper proposes a novel quantitative evaluation method that indirectly measures content preservation and style consistency through feature differences, addressing the lack of a single "golden metric" in style transfer.

Key Results

  • Qualitative: RAST produces stylized images with superior content preservation and style embedding compared to eight state-of-the-art methods (CAST, PAMA, StyTr², IEAST, ArtFlow, MCCNet, SANet, AdaIN). The paper shows examples of both artistic and photo-realistic style transfers, highlighting RAST's ability to avoid artifacts and content distortion seen in other methods.
  • Quantitative:
    • RAST achieves the highest "Content Preservation Score" (4.374) and "Style Consistency Score" (3.570) in user studies.
  • In Table 1, RAST shows competitive or superior performance across various metrics like LPIPS (content and style), content loss (Lc), and style loss (Ls).
  • While not the fastest, RAST's inference time is reasonable, with SANet being significantly faster.

Limitations and Future Work

  • Inference Time: While RAST performs well, its inference time is not as fast as some other methods like SANet. The paper notes that RAST can utilize SANet as a backbone to improve speed.
  • User Study Outliers: The paper acknowledges that IEAST and CAST had some outliers in user study results due to subtle artifacts or blurring that might not be immediately apparent.
  • Future Work: The paper doesn't explicitly detail future work, but the success of the multi-restoration approach and the novel loss functions suggests potential for further exploration in more complex style transfer scenarios or video style transfer. The ability to handle arbitrary styles and preserve content robustly opens doors for applications in creative arts, media, and beyond.

Summary updated

BibTeX
@inproceedings{ma2023rast,
  author = {Yingnan Ma and Chenqiu Zhao and Xudong Li and Anup Basu},
  title = {{RAST: Restorable Arbitrary Style Transfer via Multi-Restoration}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure