Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture

Ning Yu, Connelly Barnes, Eli Shechtman, Sohrab Amirghodsi, Michal Lukác

Abstract

This paper addresses the problem of interpolating visual textures. We formulate this problem by requiring (1) by-example controllability and (2) realistic and smooth interpolation among an arbitrary number of texture samples. To solve it we propose a neural network trained simultaneously on a reconstruction task and a generation task, which can project texture examples onto a latent space where they can be linearly interpolated and projected back onto the image domain, thus ensuring both intuitive control and realistic results. We show our method outperforms a number of baselines according to a comprehensive suite of metrics as well as a user study. We further show several applications based on our technique, which include texture brush, texture dissolve, and animal hybridization.

Introduction

Interpolating between textures—for instance, transitioning from leopard spots to tiger stripes—poses a significant challenge in computer vision. Traditional methods like $\alpha$-blending often result in "ghosting" artifacts, where patterns appear semi-transparently overlaid. Conversely, stitching methods frequently produce visible, unnatural seams. While patch-based synthesis can yield higher quality, it is computationally intensive and lacks the "by-example" controllability required for creative applications.

Texture Mixer introduces a feedforward neural network designed to map textures into a specialized latent space. Within this space, linear interpolation corresponds to perceptually smooth transitions, enabling the synthesis of realistic, hybrid textures at interactive speeds.

Methodology

The Texture Mixer architecture is built around a dual-task training regime: reconstruction and interpolation.

Dual Encoding and Latent Space

The system employs two distinct encoders to capture the essence of a texture. A global encoder ($E^g$) compresses the overall style into a $1 \times 1$ latent vector, while a local encoder ($E^l$) preserves spatial structure in a downsampled latent tensor. By interpolating these latent representations rather than raw pixels, the model avoids the pitfalls of naïve blending.

Random Shuffling

A key innovation is the random shuffling operation applied to latent tensors. To prevent the generator from simply memorizing and repeating patterns—which leads to artificial "tiling"—the network swaps adjacent small squares in the latent space. This regularizer forces the model to learn the underlying statistics of a texture, ensuring the output remains stochastic and natural.

The Texture Mixer architecture consists of a reconstruction task to ensure fidelity and an interpolation task to ensure realism.

Experiments and Results

To evaluate the model, the authors curated three high-quality datasets: Earth Texture (896 images), Animal Texture (866 images), and Plant Texture (1,074 images). These real-world RGB photos provide a more rigorous benchmark than standard synthetic datasets.

Performance and User Study

Texture Mixer achieves high-quality results in approximately 0.5 seconds, significantly faster than patch-based methods like Image Melding, which can take several minutes. In a large-scale user study on Amazon Mechanical Turk, participants consistently preferred Texture Mixer over baselines such as WCT and PSGAN. The model balances controllability, smoothness, and realism, as evidenced by its superior performance across a suite of metrics including Center Seam and Center Repetition scores.

Applications

The framework enables several creative tools, most notably the "Texture Brush." This tool allows artists to sample specific regions of a source texture and paint with an interpolated palette, as demonstrated in the synthesis of the "CVPR" text below.

Texture Mixer enables by-example control, allowing users to sample specific regions of a source texture for interpolated synthesis.

Conclusion and Limitations

While Texture Mixer excels at stochastic, natural textures, it faces limitations with strong structural textures. Rigid geometries, such as brick walls or honeycombs, can become warped or "melted" during latent shuffling and interpolation. Future research may explore incorporating geometric constraints to better preserve these structured patterns while maintaining the benefits of neural interpolation.

Summary updated

Abstract

This paper addresses the problem of interpolating visual textures. We formulate this problem by requiring (1) by-example controllability and (2) realistic and smooth interpolation among an arbitrary number of texture samples. To solve it we propose a neural network trained simultaneously on a reconstruction task and a generation task, which can project texture examples onto a latent space where they can be linearly interpolated and projected back onto the image domain, thus ensuring both intuitive control and realistic results. We show our method outperforms a number of baselines according to a comprehensive suite of metrics as well as a user study. We further show several applications based on our technique, which include texture brush, texture dissolve, and animal hybridization.
BibTeX
@inproceedings{yu2019texture,
  author = {Ning Yu and Connelly Barnes and Eli Shechtman and Sohrab Amirghodsi and Michal Lukác},
  title = {{Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=-6382YeBKvM},
}
Figure