Abstract: The real world exhibits an abundance of non-stationary textures. Examples include textures with large-scale structures, as well as spatially variant and inhomogeneous textures. While existing example-based texture synthesis methods can cope well with stationary textures, non-stationary textures still pose a considerable challenge, which remains unresolved. In this paper, we propose a new approach for example-based non-stationary texture synthesis. Our approach uses a generative adversarial network (GAN), trained to double the spatial extent of texture blocks extracted from a specific texture exemplar. Once trained, the fully convolutional generator is able to expand the size of the entire exemplar, as well as of any of its sub-blocks. We demonstrate that this conceptually simple approach is highly effective for capturing large-scale structures, as well as other non-stationary attributes of the input exemplar. As a result, it can cope with challenging textures, which, to our knowledge, no other existing method can handle.
The Challenge of Non-Stationary Textures
Example-based texture synthesis is a classic computer graphics problem: given a small sample of a texture, generate a larger version of it. The field has largely solved "stationary" textures (e.g., uniform grass, gravel, or stucco) where local statistics remain constant across the image.
However, "non-stationary" textures remain a massive challenge. These are textures with large-scale irregular structures, spatial variations, or global trends—think of the concentric rings of a tree trunk, the diverging veins of a leaf, or the perspective shift in a photo of a brick wall. Existing methods, whether classical patch-stitching or recent deep learning approaches like Gram-matrix matching, usually fail on these. They tend to produce repetitive, periodic artifacts or completely lose the overarching global structure of the exemplar.
Methodology: Adversarial Expansion
The authors frame texture synthesis as a self-supervised "adversarial expansion" problem. Instead of matching global statistics, they train a Generative Adversarial Network (GAN) specifically on a single input texture exemplar to learn how it grows.
- The Generator ($G$): A fully convolutional network (utilizing residual blocks) takes a $k \times k$ crop from the exemplar and learns to expand it into a $2k \times 2k$ block.
- The Discriminator ($D$): A PatchGAN discriminator evaluates $2k \times 2k$ blocks and tries to classify if they are real (cropped directly from the larger original exemplar) or fake (expanded by the generator).
- Loss Function: The network optimizes a combination of adversarial loss, an $L_1$ pixel loss, and a VGG-19 style loss to stabilize training and ensure structural integrity.
Because the generator is fully convolutional, it is not restricted to the training crop size. Once trained, it can be fed the entire original exemplar to double its overall size, or fed its own output to expand the texture indefinitely.
Key Insights and Results
A critical distinction of this method is that it goes beyond simple super-resolution. While super-resolution sharpens existing pixels, this generator learns to inject entirely new content—such as drawing brand new wood rings or leaf veins—that respects the spatial trends and spacing of the input. Visualizations of the network's internal feature maps reveal that the chain of residual blocks is specifically responsible for replicating and spatially transforming structures to create these new large-scale patterns.
Performance Highlights
- State-of-the-Art Visuals: The method successfully maintains global structures (like radial wood rings and directional feathers) where competing methods often devolve into messy, stationary noise.
- Blazing Fast Inference: Once trained, doubling a 600x400 texture requires only a single forward pass, taking a mere 4-5 milliseconds on an NVIDIA Titan Xp GPU.
- High Stability: The method passes rigorous stress tests where the output is repeatedly fed back into the generator. It can expand a 64x64 patch to a 2048x2048 image (a 32x expansion) over five cycles without losing sharpness.
Limitations and Future Work
Despite its success, the approach requires a dedicated model to be trained from scratch for each texture exemplar, taking roughly 2 to 5 hours. Border artifacts occasionally appear due to convolution padding, and the model can fail if an exemplar contains a unique, non-repeating structure (a "singularity") like the center of a sunflower. Future work may focus on training on multiple textures of the same class to build more generalized models.