When we think of a texture—like a patch of sand, a field of grass, or a gravel path—we instinctively assume that its patterns repeat uniformly. In computer vision, this intuition is formalized as the spatial stationarity assumption. It posits that the statistical properties of an image are invariant across different spatial locations; any small window cropped from the texture should look statistically identical to any other.
For decades, texture synthesis algorithms relied entirely on this assumption. Early statistical methods and deep learning frameworks excelled at taking a small exemplar of pebbles and expanding it into an infinite, seamless canvas. However, real-world surfaces rarely obey perfect stationarity. Consider the concentric rings of a tree trunk, the branching veins of a leaf, the perspective shift of a receding road, or the strict geometric layout of a brick wall. These are non-stationary and structured textures. Their visual properties change dramatically depending on where you look, and their elements maintain rigid, long-range spatial relationships.
Synthesizing these patterns requires moving beyond simple statistical replication to architectures that can simultaneously respect global structural constraints and generate natural local variations.
Why Global Statistics Fail
The limitations of traditional texture synthesis became glaringly obvious with the advent of deep style transfer. A seminal paper by Leon A. Gatys demonstrated that the "style" of an image could be captured by calculating the correlations between feature maps across different layers of a pre-trained Convolutional Neural Network (CNN), stored as a Gram matrix.
While Gram matrices are incredibly effective at capturing stochastic textures (like the brushstrokes of Van Gogh), they completely discard spatial coordinates. As Gang Liu et al. observed, representing textures solely through global statistical correlations fails to encode the rigid geometric alignment of quasi-periodic patterns. Optimization frameworks that rely on average statistical expectations—such as the macrocanonical models discussed by Valentin De Bortoli et al.—prioritize global averages over hard spatial constraints.
Gram Matrix / Statistical Averaging:
[Feature Map A] ⊗ [Feature Map B] ---> Global Correlation (Spatial Metadata Lost)
Result: Excellent for sand/noise; destroys brick walls and leaf veins.
When a model discards spatial metadata, it loses the ability to understand where a feature belongs relative to the rest of the image. As a result, trying to synthesize a non-stationary texture like a brick wall or a structured fabric using global style encoders causes the pattern to lose its global coherence, melting into a chaotic, disjointed soup of local textures (Jun-Yan Zhu et al.). Conversely, feed-forward networks that lack long-range spatial dependencies often swing to the opposite extreme, generating robotic, overly uniform repetitions that lack natural variation (Shi Wu et al.).
The Spectrum of Regularity
To solve this, researchers treat textures not as a binary choice between "random noise" and "perfect grid," but as a continuous spectrum of regularity.
As established by James Hays et al., most real-world textures occupy a middle ground known as near-regular textures (NRTs). These patterns possess a clear underlying geometric lattice but feature distinct local variations. A brick wall has a predictable grid, but every individual brick has unique cracks, color gradients, and weathering.
[Purely Stochastic] <-------------------- [Near-Regular] --------------------> [Strictly Periodic]
(Clouds, Rust) (Brick Walls) (Digital Grids)
No global structure; Global lattice + Perfect repetition;
Easy to synthesize Local variations No natural variation
If a synthesis model enforces the global grid too rigidly, the result looks artificial. If it ignores the grid, the structure collapses. Bowei Chen et al. addressed this by using coordinate-based neural networks to model near-periodic patterns, combining periodicity-aware coordinate warping with patch-based losses. This allows the network to adhere to a global layout while preserving the unique, non-periodic details that make a texture look authentic.
Mechanisms for Overcoming Stationarity
Modern non-stationary texture synthesis relies on mechanisms that explicitly anchor local appearance to global spatial context. Rather than assuming a uniform distribution of features, these models use structural guides, multi-scale pyramids, and feature injection.
1. Spatial Conditioning and Structural Anchors
One way to synthesize non-stationary textures is to make the generation process dependent on an underlying geometric guide. For instance, Shuai Yang et al. tackled the problem of transferring highly non-stationary artistic text effects (such as flames or frayed denim). They achieved this by calculating a target shape's "skeleton" and linking local appearance features directly to their distance from this central structure. This ensures that internal gradients and edge textures vary predictably based on their exact spatial position.
On a macro scale, Ivan Skorokhodov et al. bypassed the repetitive limitations of traditional texture synthesis by anchoring latent codes directly to continuous spatial coordinates, allowing complex semantic landscapes to evolve realistically across an infinite horizon.
2. Multi-Scale Internal Pyramids
When learning from a single non-stationary exemplar, models must capture both macro-scale trends (like the curvature of wood rings) and micro-scale details (the grain of the wood). Tamar Rott Shaham et al. introduced SinGAN, a multi-scale pyramid of Generative Adversarial Networks (GANs) trained on a single natural image. The coarsest level of the pyramid learns the global layout and spatial arrangements of the scene, while the finer levels capture the local texture.
Similarly, the fully convolutional approach explored in Non-Stationary Texture Synthesis by Adversarial Expansion trains a generator to double the spatial extent of random crops from a single image. Because the network is forced to maintain structural continuity across the expanded boundaries, it learns to propagate global trends—like leaf veins—without creating repetitive artifacts.
Multi-Scale Generation Pyramid:
[Coarse Scale GAN] ---> Captures Global Layout & Trends (e.g., Wood Ring Curvature)
↓
[Medium Scale GAN] ---> Captures Intermediate Structures
↓
[Fine Scale GAN] ---> Generates Micro-Details & Textures (e.g., Fine Wood Grain)
3. Diffusion and Self-Rectification
The latest frontier leverages the generative power of denoising diffusion probabilistic models. While standard diffusion models excel at generating objects, they can struggle with the precise, user-guided expansion of non-stationary textures.
To bridge this gap, Yang Zhou et al. introduced a "self-rectification" framework. When a user provides a rough, incoherent sketch of a non-stationary texture, the model uses feature injection during both the inversion and sampling stages of the diffusion process. By injecting self-attention features from a reference exemplar into the generation path of the sketch, the model forces the output to preserve the user's large-scale custom structure while faithfully dressing it in the intricate, non-stationary details of the target material.
Redefining Evaluation
As the synthesis paradigms shifted, traditional evaluation metrics also had to evolve. Historically, texture quality assessment algorithms assumed spatial homogeneity, measuring how closely the statistical properties of a synthesized image matched the original across random patches. However, as Dengxin Dai et al. pointed out, metrics based purely on homogeneity fail to capture the true synthesizability of complex patterns.
To evaluate non-stationary textures where visual characteristics change continuously across space, S. Alireza Golestaneh developed a training-free metric (IGSTQA). By analyzing spatial and statistical attributes across multiple scales in both the image and gradient magnitude domains, this approach successfully quantifies structural fidelity and local variation simultaneously. This provides a mathematical framework for evaluating texture quality that closely aligns with human visual perception, ensuring that the field can reliably measure progress as models continue to break free from the stationarity assumption.
Key Foundational Work
Quantitative Evaluation of Near Regular Texture Synthesis Algorithms (2006) — contributed 2 concept elements to this direction.
Texture Segmentation by Multiscale Aggregation of Filter Responses and Shape Elements (2003) — contributed 1 concept elements to this direction.