Texture Synthesis with Recurrent Variational Auto-Encoder

Rohan Chandra, Sachin Grover, Kyungjun Lee, Moustafa Meshry, Ahmed TahaarXiv 2017 · arXiv

Abstract

Texture synthesis involves generating large, natural-looking patterns from small input samples. This paper investigates the use of the Deep Recurrent Attentive Writer (DRAW), a recurrent variational auto-encoder, for this task. The authors propose the FLTBNK loss function, which utilizes filter bank responses to capture structural features and ensure rotational invariance. By training independent networks to generate neighboring tiles, the approach achieves seamless texture expansion. Evaluation on the Describable Textures Dataset (DTD) shows that while VAE-based models often produce blurry outputs, the FLTBNK loss provides superior structural consistency and reliability compared to traditional pixel-wise metrics.

Introduction

Texture synthesis is the process of taking a small "seed" image—such as a patch of grass or a brick wall—and generating a larger, non-repeating version that appears natural. This task presents two primary challenges: ensuring smooth alignment between stitched tiles to avoid visible seams, and maintaining the structural integrity of the underlying pattern (e.g., line orientation or grain texture) rather than merely matching colors. While deep learning models like GANs and VGG-based optimization have shown promise, they often require large inputs or significant resources. This paper addresses these issues by generating high-quality neighboring tiles from very small (28x28 pixel) inputs using a generative model that learns local pixel correlations.

Methodology

The proposed pipeline is built around the Deep Recurrent Attentive Writer (DRAW) network. DRAW is a recurrent variational auto-encoder (VAE) that employs an iterative attention mechanism to "read" from and "write" to an image over several time steps.

The DRAW network architecture, illustrating the recurrent encoder-decoder structure and the iterative attention mechanism.

Instead of generating a large image in a single pass, the researchers trained four independent DRAW networks. Each network is specialized to generate one neighboring tile (Up, Down, Left, or Right) given a central seed tile. These generated tiles are then stitched together to expand the texture seamlessly.

The FLTBNK Loss Function

A key innovation of this work is the FLTBNK (Filter Bank) loss. Standard loss functions like L2 (mean squared error) often produce blurry results because they average pixel values. To better capture the "essence" of a texture, the authors utilized the Leung-Malik (LM) filter bank, consisting of 48 filters (edge, bar, and spot filters at various scales and orientations).

Unlike traditional methods that use non-differentiable histograms, this approach computes loss directly from filter responses, enabling backpropagation. To address noise and color consistency, the authors supplemented the filter bank loss with Total Variation (TV) loss and a color regularizer to match the mean color of the original seed.

Experimental Setup

The model was evaluated using the Describable Textures Dataset (DTD), which contains diverse texture patterns categorized by human descriptions (e.g., "striped," "woven"). The study focused on five regular texture patterns for primary training and evaluation.

Results

The authors compared the performance of the FLTBNK loss against several baselines, including standard Bernoulli and L2 losses, as well as VGG-based Gram-matrix losses.

Comparison of various loss functions evaluated in the study, highlighting the proposed FLTBNK loss alongside qualitative results.

Qualitative and Quantitative Performance

Qualitatively, the FLTBNK loss outperformed L2 in capturing complex outlines and geometric patterns, such as those found in chessboard-like textures. Quantitatively, while VGG-based losses (popularized by style transfer techniques) are powerful, they often fail on small 28x28 inputs because the VGG architecture's pooling layers discard critical spatial information at low resolutions. The FB-trained network achieved a smaller Texture Histogram distance, which is considered more informative for texture-specific similarity than VGG distance.

Human Evaluation

A study involving 18 participants using a 5-point Likert scale revealed that while traditional losses (Cross-entropy, L2) occasionally produced "Excellent" results, they were inconsistent. In contrast, FLTBNK received the highest number of "Very Good" ratings, indicating it is a more reliable loss function for maintaining consistent texture quality across different patterns.

Discussion and Conclusion

Despite its successes, the model shares a common limitation with other VAE-based approaches: the outputs tend to be somewhat blurry due to the optimization of a unimodal distribution. Additionally, the current color regularization focuses on independent R, G, and B channels, occasionally leading to slight color shifts because it lacks a deep understanding of inter-channel correlations.

Future work may involve developing a more unified model for multi-directional expansion to reduce the complexity of training four separate networks. Nonetheless, this research demonstrates that repurposing recurrent VAEs with specialized filter-bank losses provides a robust framework for small-scale texture synthesis.

Summary updated

Abstract

We propose a recurrent variational auto-encoder for texture synthesis. A novel loss function, FLTBNK, is used for training the texture synthesizer. It is rotational and partially color invariant loss function. Unlike L2 loss, FLTBNK explicitly models the correlation of color intensity between pixels. Our texture synthesizer generates neighboring tiles to expand a sample texture and is evaluated using various texture patterns from Describable Textures Dataset (DTD). We perform both quantitative and qualitative experiments with various loss functions to evaluate the performance of our proposed loss function (FLTBNK) --- a mini-human subject study is used for the qualitative evaluation.
BibTeX
@misc{chandra2017texture,
  author = {Rohan Chandra and Sachin Grover and Kyungjun Lee and Moustafa Meshry and Ahmed Taha},
  title = {{Texture Synthesis with Recurrent Variational Auto-Encoder}},
  year = {2017},
  eprint = {1712.08838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.08838},
}
Figure