Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture

Ning Yu, Connelly Barnes, Eli Shechtman, Sohrab Amirghodsi, Michal Lukác

Abstract

This paper introduces Texture Mixer, a feedforward neural network designed for the smooth interpolation of visual textures. By formulating the problem with a focus on by-example controllability and realistic transitions, the authors propose a system trained simultaneously on reconstruction and generation. The model projects texture samples into a shared latent space where linear interpolation yields realistic, artifact-free results. Evaluated against a suite of metrics and user studies, Texture Mixer demonstrates superior performance and speed, enabling applications like texture brushes and animal hybridization.

Introduction

In digital content creation, transitioning between materials—such as blending grass into dirt or merging different animal furs—is a frequent requirement. Traditional methods often fall short: "alpha blending" results in "ghosting" artifacts, while non-parametric patch-based synthesis is computationally expensive and often fails to capture complex natural statistics. While Generative Adversarial Networks (GANs) offer high-quality synthesis, they typically lack the fine-grained controllability needed for specific artistic placement.

Methodology

The Texture Mixer architecture is designed to bridge the gap between realism and control. It is trained on two concurrent objectives: a reconstruction task to ensure fidelity to input samples, and an interpolation task to ensure the latent space between samples represents valid textures.

Dual-Encoder Architecture

The system utilizes two distinct encoders to decompose texture information:

  • Global Encoder ($E^g$): Extracts a $1 \times 1$ vector representing the overall style and statistical properties.
  • Local Encoder ($E^l$): Produces a latent tensor that captures spatial structures at a reduced resolution.
The Texture Mixer architecture trained on simultaneous reconstruction and interpolation tasks.

Latent Shuffling

A key innovation is the random shuffling operation applied within the latent space. By swapping adjacent squares of the latent tensor at multiple scales, the network breaks up repetitive "tiling" patterns. This stochasticity ensures that the final generator ($G$) produces natural-looking textures even when covering large areas.

Datasets and Training

To support this research, the authors curated two high-density datasets:

  • Earth Texture Dataset: 994 images of natural terrains like soil and gravel.
  • Animal Texture Dataset: 961 images of diverse animal furs and skins.

These datasets provide the necessary variety to train a robust interpolation manifold, allowing the network to learn the underlying logic of texture transitions.

Results and Applications

Texture Mixer achieves high-quality results that avoid the common pitfalls of previous methods. Quantitatively, it outperforms baselines like PSGAN and AdaIN across metrics for controllability and realism. In a user study, participants preferred Texture Mixer's results with rates often exceeding 80%.

A smooth interpolation sequence showing the transition between two distinct natural textures.

Beyond simple blending, the model supports advanced applications. Because it is a feedforward network, it can generate $1024 \times 1024$ textures in roughly 0.5 seconds. This speed enables interactive tools like "texture brushes" and "texture dissolve" effects.

Conclusion

Texture Mixer provides a powerful framework for interpolating textures by mapping them to a continuous, controllable manifold. While currently trained on specific categories, it sets a new standard for speed and quality in texture synthesis. Future work may extend this approach to video textures and 3D surface properties, further expanding the toolkit for digital artists.

Summary updated

Abstract

This paper addresses the problem of interpolating visual textures. We formulate this problem by requiring (1) by-example controllability and (2) realistic and smooth interpolation among an arbitrary number of texture samples. To solve it we propose a neural network trained simultaneously on a reconstruction task and a generation task, which can project texture examples onto a latent space where they can be linearly interpolated and projected back onto the image domain, thus ensuring both intuitive control and realistic results. We show our method outperforms a number of baselines according to a comprehensive suite of metrics as well as a user study. We further show several applications based on our technique, which include texture brush, texture dissolve, and animal hybridization.
BibTeX
@inproceedings{yu2019texture,
  author = {Ning Yu and Connelly Barnes and Eli Shechtman and Sohrab Amirghodsi and Michal Lukác},
  title = {{Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=-6382YeBKvM},
}
Figure