Introduction
Texture synthesis is a fundamental task in computer vision, yet it remains challenging to balance speed with structural integrity. Traditional methods typically fall into two categories: Single-Texture Synthesis (STS), which produces high-quality results but requires hours of training per image, and Multi-Texture Synthesis (MTS), which is fast but often fails to preserve the global structure of "non-stationary" textures—those with large-scale organized patterns like spiderwebs or leaf veins.
The Challenge of Non-Stationary Textures
The primary issue with existing MTS models is their inability to capture long-range dependencies. When generating a texture like a cobweb, a standard model might produce the local "sticky" lines but fail to connect them into a coherent radial structure. This paper introduces MTGAN, a multi-scale generative model designed to bridge this gap by learning within specific texture domains.
Methods: The MTGAN Architecture
To address the structural limitations of previous models, the authors proposed a novel architecture that operates across multiple scales simultaneously.
Multi-Scale Processing
MTGAN employs three parallel branches at 4x, 8x, and 16x down-sampling. This design allows the model to capture fine-grained details in the high-resolution branch while maintaining a large receptive field in the lower-resolution branches to understand the "big picture" structural grammar.
Layer-wise Feature Fusion
Instead of a simple concatenation at the end of the network, MTGAN uses Fusion Blocks at every residual block. Information flows from the coarser scales (secondary branches) to the finer scales (primary branch) in a bottom-up fashion. This ensures that the global layout consistently guides the generation of local details.
Category-Specific Training and Metrics
The model is trained on specific categories (e.g., "veined" or "cobwebbed") from the Describable Texture Dataset (DTD). This "single domain" approach allows the model to learn the underlying structural rules common to a class of textures. Furthermore, the authors introduced the Multi-Scale Texture Similarity (MSTS) metric, which evaluates texture quality based on deep feature distributions across scales, providing a more accurate measure of human perception than standard metrics like FID.
Key Results and Efficiency
MTGAN demonstrates a significant leap in both quality and performance:
- Speed: It is 5x to 27x faster than state-of-the-art STS methods like SinGAN. Once a category is learned, fine-tuning for a specific unseen image takes only minutes.
- Structural Fidelity: In user studies, MTGAN was preferred for its ability to expand complex textures without the "scrambling" effect common in other MTS models.
- Quantitative Performance: On the DTD-11 dataset, MTGAN achieved an MSTS score of 156.66, outperforming the next best MTS method (Neural FFTs) by a wide margin.
Conclusion
By combining multi-scale parallel processing with category-specific learning, MTGAN successfully synthesizes high-quality non-stationary textures within a single domain. This approach offers a practical middle ground between the slow, high-fidelity STS models and the fast but structurally weak MTS models, opening new possibilities for efficient texture-aware image generation.
Summary updated