Texture Reformer: Towards Fast and Universal Interactive Texture Transfer

Zhizhong Wang, Lei Zhao, Haibo Chen, Ailin Li, Zhiwen Zuo, Wei Xing, Dongming Lu

Abstract

Texture Reformer is a fast and universal neural-based framework for interactive texture transfer with user-specified guidance. It addresses the challenges of task diversity, guidance map simplicity, and execution efficiency through a novel feed-forward multi-view and multi-stage synthesis procedure. The framework consists of three stages: global view structure alignment, local view texture refinement, and holistic effect enhancement. Additionally, a learning-free view-specific texture reformation (VSTR) operation with a new semantic map guidance strategy is introduced to achieve accurate semantic-guided and structure-preserved texture transfer. Experimental results demonstrate that Texture Reformer is 2-5 orders of magnitude faster than state-of-the-art interactive texture transfer algorithms while achieving higher quality results.

Introduction

Interactive texture transfer allows users to apply the visual style of a source image to a target using simple semantic maps (color-coded masks). This technology is vital for applications like turning doodles into artwork, swapping clothing patterns, or transferring text effects. However, existing methods often struggle with a "trilemma": they are either task-specific, fail to preserve complex structures from simple doodles, or are too slow for real-time interaction due to iterative optimization.

Overview of Texture Reformer applications: (a) Doodles-to-artworks, (b) Texture Pattern Editing, (c) Text Effects Transfer, and (d) Virtual Clothing Manipulation.

Methodology

Texture Reformer replaces slow optimization with a fast, three-stage feed-forward pipeline:

  1. Stage I: Global View Structure Alignment: Utilizing deep layers of a pre-trained VGG network and large patch sizes, the model performs global alignment. This ensures the large-scale structures of the source texture match the target's overall shape.
  2. Stage II: Local View Texture Refinement: By moving to shallower layers and smaller patches, the model refines details, correcting warping from the first stage and ensuring fine-grained consistency.
  3. Stage III: Holistic Effect Enhancement: A learning-free Statistics-based Enhancement (SE) matches low-level traits like color and contrast.

A core innovation is the View-Specific Texture Reformation (VSTR) operation. VSTR projects semantic maps and image features into a common space, overcoming the information discrepancy between raw pixels and deep neural features to ensure accurate, structure-preserving matching.

Example of the doodles-to-artworks task, where a simple semantic doodle is transformed into a high-quality portrait based on a source painting.

Datasets and Evaluation

The framework utilizes pre-trained VGG-19 features, with an auto-encoder trained on the Microsoft COCO dataset. For evaluation, the authors used images from benchmarks such as DeepFashion (for clothing manipulation) and Neural Doodle, as well as various artistic paintings and stylized text images.

Key Results and Insights

The framework's "learning-free" nature makes it universal, capable of processing diverse textures without specific training. By separating structure alignment (Global View) from detail refinement (Local View), it achieves high visual fidelity.

  • Efficiency: Texture Reformer processes 512x512 images in approximately 0.95 seconds on a GPU—100 to 100,000 times faster than optimization-based methods like Gatys2017 or STROTSS.
  • Quality: The model outperformed competitors in SSIM (Structural Similarity) and LPIPS (Perceptual Loss) metrics. In a user study with 50 participants, Texture Reformer's results were preferred 32.1% of the time, beating the next best competitor (CFITT) at 26.2%.

Limitations and Future Work

Despite its speed, the model has limitations. It currently lacks support for patch rotation or scaling, which can cause artifacts if the target guidance is a transformed version of the source. Additionally, very complex decorative elements (like eyes or hands in cartoons) may not transfer perfectly with simple semantic maps. Future directions include integrating geometric transformations or lighting constraints to further improve realism.

Summary updated

Abstract

In this paper, we present the texture reformer, a fast and universal neural-based framework for interactive texture transfer with user-specified guidance. The challenges lie in three aspects: 1) the diversity of tasks, 2) the simplicity of guidance maps, and 3) the execution efficiency. To address these challenges, our key idea is to use a novel feed-forward multi-view and multi-stage synthesis procedure consisting of I) a global view structure alignment stage, II) a local view texture refinement stage, and III) a holistic effect enhancement stage to synthesize high-quality results with coherent structures and fine texture details in a coarse-to-fine fashion. In addition, we also introduce a novel learning-free view-specific texture reformation (VSTR) operation with a new semantic map guidance strategy to achieve more accurate semantic-guided and structure-preserved texture transfer. The experimental results on a variety of application scenarios demonstrate the effectiveness and superiority of our framework. And compared with the state-of-the-art interactive texture transfer algorithms, it not only achieves higher quality results but, more remarkably, also is 2-5 orders of magnitude faster. Code is available at https://github.com/EndyWon/Texture-Reformer.
BibTeX
@inproceedings{wang2022texture,
  author = {Zhizhong Wang and Lei Zhao and Haibo Chen and Ailin Li and Zhiwen Zuo and Wei Xing and Dongming Lu},
  title = {{Texture Reformer: Towards Fast and Universal Interactive Texture Transfer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=KvvrN4fneW},
}
Figure