Abstract
Texture Reformer is a fast and universal neural-based framework for interactive texture transfer with user-specified guidance. It addresses the challenges of task diversity, guidance map simplicity, and execution efficiency through a novel feed-forward multi-view and multi-stage synthesis procedure. The framework consists of three stages: global view structure alignment, local view texture refinement, and holistic effect enhancement. Additionally, a learning-free view-specific texture reformation (VSTR) operation with a new semantic map guidance strategy is introduced to achieve accurate semantic-guided and structure-preserved texture transfer. Experimental results demonstrate that Texture Reformer is 2-5 orders of magnitude faster than state-of-the-art interactive texture transfer algorithms while achieving higher quality results.
Introduction
Interactive texture transfer allows users to apply the visual style of a source image to a target using simple semantic maps (color-coded masks). This technology is vital for applications like turning doodles into artwork, swapping clothing patterns, or transferring text effects. However, existing methods often struggle with a "trilemma": they are either task-specific, fail to preserve complex structures from simple doodles, or are too slow for real-time interaction due to iterative optimization.
Methodology
Texture Reformer replaces slow optimization with a fast, three-stage feed-forward pipeline:
- Stage I: Global View Structure Alignment: Utilizing deep layers of a pre-trained VGG network and large patch sizes, the model performs global alignment. This ensures the large-scale structures of the source texture match the target's overall shape.
- Stage II: Local View Texture Refinement: By moving to shallower layers and smaller patches, the model refines details, correcting warping from the first stage and ensuring fine-grained consistency.
- Stage III: Holistic Effect Enhancement: A learning-free Statistics-based Enhancement (SE) matches low-level traits like color and contrast.
A core innovation is the View-Specific Texture Reformation (VSTR) operation. VSTR projects semantic maps and image features into a common space, overcoming the information discrepancy between raw pixels and deep neural features to ensure accurate, structure-preserving matching.
Datasets and Evaluation
The framework utilizes pre-trained VGG-19 features, with an auto-encoder trained on the Microsoft COCO dataset. For evaluation, the authors used images from benchmarks such as DeepFashion (for clothing manipulation) and Neural Doodle, as well as various artistic paintings and stylized text images.
Key Results and Insights
The framework's "learning-free" nature makes it universal, capable of processing diverse textures without specific training. By separating structure alignment (Global View) from detail refinement (Local View), it achieves high visual fidelity.
- Efficiency: Texture Reformer processes 512x512 images in approximately 0.95 seconds on a GPU—100 to 100,000 times faster than optimization-based methods like Gatys2017 or STROTSS.
- Quality: The model outperformed competitors in SSIM (Structural Similarity) and LPIPS (Perceptual Loss) metrics. In a user study with 50 participants, Texture Reformer's results were preferred 32.1% of the time, beating the next best competitor (CFITT) at 26.2%.
Limitations and Future Work
Despite its speed, the model has limitations. It currently lacks support for patch rotation or scaling, which can cause artifacts if the target guidance is a transformed version of the source. Additionally, very complex decorative elements (like eyes or hands in cartoons) may not transfer perfectly with simple semantic maps. Future directions include integrating geometric transformations or lighting constraints to further improve realism.
Summary updated