Texture synthesis has long been divided into two worlds. On one side are stochastic textures—patterns like sand, grass, or pebbles where the visual identity is defined by a statistical distribution. On the other are structured and non-stationary textures—patterns like brick walls, wood grain, or building facades where global geometry and long-range trends are just as important as local color.
While early deep learning methods mastered the stochastic world, they famously struggled with structure. Modern research has shifted toward breaking the "stationarity assumption" to capture the rigid regularities and evolving trends of the real world.
The Failure of Statistical Averaging
The breakthrough in neural texture synthesis began with Leon Alexander Gatys, who demonstrated in Image Style Transfer Using Convolutional Neural Networks that the correlations between features in a pre-trained CNN (stored in a Gram matrix) could represent "style."
However, this approach relies on a spatial average. By collapsing spatial information into a single matrix, the model assumes stationarity: the idea that the texture's properties are the same everywhere. As noted in Incorporating long-range consistency in CNN-based texture generation, this works for clouds but fails miserably for bricks or text. Without a mechanism to track spatial offsets, the model cannot maintain the geometric consistency or periodic alignment required for man-made structures.
Capturing Non-Stationary Trends
Many natural textures are not just structured; they are non-stationary, meaning their patterns change gradually across space. Think of the concentric growth rings of a tree or the branching veins of a leaf.
Traditional statistical methods treat these as errors to be averaged out. To solve this, Yang Zhou and colleagues proposed Non-stationary texture synthesis by adversarial expansion. Instead of sampling random patches, they framed synthesis as a "spatial expansion" task. By training a generator to double the size of a patch, the model learns the underlying growth logic—the "DNA" of how a vein branches or a ring curves—rather than just repeating local pixels. This prevents the fragmentation of large-scale features that occurs when models assume every part of the image is statistically identical.
The Geometry of Near-Periodic Patterns
Between perfectly random noise and perfectly rigid grids lies the domain of Near-Regular Textures (NRTs). These are patterns like woven fabrics or masonry where a clear repeating lattice exists, but every unit has slight, natural variations.
Research by James Hays in Quantitative Evaluation of Near Regular Texture Synthesis Algorithms highlighted that general-purpose synthesis often disrupts these underlying structural rules. To fix this, newer models like NPP-Net, developed by Bowei Chen and Tiancheng Zhi, use Learning Continuous Implicit Representation for Near-Periodic Patterns. They combine "periodicity-aware coordinate warping" with patch-based losses to ensure the model respects the global grid while still allowing for the unique local details that make a texture look authentic.
Implicit Fields and Coordinate-Based Synthesis
The most recent frontier involves moving away from pixel grids entirely. Researchers like Philipp Henzler and Tobias Ritschel have explored Learning a Neural 3D Texture Space From 2D Exemplars, using Multi-Layer Perceptrons (MLPs) to map coordinates to colors.
While these coordinate-based models are excellent for "infinite zoom" and smooth interpolation, they initially struggled with the same rigid regularities as early CNNs. The solution, as seen in the Implicit Periodic Field Network (IPFN) by Haiwei Chen, is to bake periodicity directly into the network's architecture. By using learnable periodic encodings, these models can synthesize high-quality, tileable patterns that maintain structural rhythm across any scale.
Why It Works: From Statistics to Semantics
The shift in this field is a transition from low-level statistics to high-level structural awareness.
- Stochastic models work by matching the "what" (the colors and edges).
- Structured models work by understanding the "where" (the lattice and the grid).
- Non-stationary models work by understanding the "how" (the growth trends and perspective shifts).
By combining these perspectives—using adversarial expansion for trends, implicit fields for scale, and lattice-awareness for regularity—researchers have moved beyond simple "noise" to generate digital surfaces that respect the complex, non-random logic of the physical world.