Abstract
In this paper, we present the texture reformer, a fast and universal neural-based framework for interactive texture transfer with user-specified guidance. The challenges lie in three aspects: 1) the diversity of tasks, 2) the simplicity of guidance maps, and 3) the execution efficiency. To address these challenges, our key idea is to use a novel feed-forward multi-view and multi-stage synthesis procedure consisting of I) a global view structure alignment stage, II) a local view texture refinement stage, and III) a holistic effect enhancement stage to synthesize high-quality results with coherent structures and fine texture details in a coarse-to-fine fashion. In addition, we also introduce a novel learning-free view-specific texture reformation (VSTR) operation with a new semantic map guidance strategy to achieve more accurate semantic-guided and structure-preserved texture transfer. The experimental results on a variety of application scenarios demonstrate the effectiveness and superiority of our framework. And compared with the state-of-the-art interactive texture transfer algorithms, it not only achieves higher quality results but, more remarkably, also is 2-5 orders of magnitude faster.
Introduction
Interactive texture transfer is the process of migrating the visual characteristics of a source texture onto a target image based on user-defined semantic guidance, such as color-coded doodles or masks. This technology is essential for creative workflows, including digital painting, textile design, and virtual try-on applications. Despite its potential, existing solutions often face a "trilemma": they are either specialized for specific tasks, fail to preserve complex structures from simple user inputs, or require slow, iterative optimization processes that hinder real-time interactivity.
Methodology
The proposed Texture Reformer framework addresses these challenges through a feed-forward, learning-free approach that replaces optimization with a three-stage "coarse-to-fine" synthesis procedure.
Multi-Stage Synthesis Procedure
- Stage I: Global View Structure Alignment: By utilizing deep layers of a pre-trained VGG network and large patch sizes, the system performs a global alignment to match the broad structural layout of the source texture with the target guidance map.
- Stage II: Local View Texture Refinement: This stage uses shallower layers and a small 3x3 patch size to rectify and refine the results of Stage I, focusing on fine-grained details and correcting any structural warping.
- Stage III: Holistic Effect Enhancement: To handle low-level features like color, brightness, and contrast, the framework employs Statistics-based Enhancement (SE) on the shallowest layers.
View-Specific Texture Reformation (VSTR)
A key innovation is the View-Specific Texture Reformation (VSTR) operation. Unlike previous methods that simply stacked semantic maps onto RGB images, VSTR projects both the textures and the semantic guidance into a shared VGG embedding space. This allows for more accurate semantic matching and better preservation of the source texture's internal structure.
Results and Evaluation
The framework is inherently universal because the core VSTR operation is an algorithm rather than a set of trained weights. While the auto-encoder/decoder used for image reconstruction was trained on the Microsoft COCO dataset, the texture transfer logic itself does not require task-specific training.
Efficiency and Speed
Texture Reformer is 2 to 5 orders of magnitude faster than previous state-of-the-art methods. For a 512x512 image, it achieves a processing time of approximately 0.95 seconds on a GPU. In contrast, optimization-based methods like Neural Doodle can take several minutes to produce a single result.
Quality and User Preference
In a user study involving 50 participants, Texture Reformer's results were preferred 32.1% of the time, outperforming the next best competitor, CFITT (26.2%), and significantly beating neural-based predecessors like Neural Doodle (10.3%). Qualitative results demonstrate that the model excels at preserving recognizable features—such as facial details in portraits or intricate textile patterns—avoiding the "texture mush" often seen in Gram-matrix-based style transfer.
Conclusion and Future Work
Texture Reformer provides a fast, universal, and high-quality solution for interactive texture transfer. By leveraging multi-view perception and semantic fusion in feature space, it overcomes the efficiency and structural limitations of prior work. Future research will explore optimizing VSTR for shallower network layers and extending the framework to handle temporal consistency in video texture transfer.
Summary updated