Neural Texture Synthesis with Guided Correspondence

Yang Zhou, Kaijian Chen, Rongjun Xiao, Hui Huang

Abstract

Texture synthesis aims to generate large, seamless images from small source examples. While traditional Markov Random Field (MRF) methods preserve structure well, they often lack the flexibility of neural approaches. Conversely, neural methods frequently produce blurry or repetitive results. This paper introduces a "Guided Correspondence" framework that integrates MRF-style patch similarity with CNN representations. By employing a Guided Correspondence Distance (GCD) and a specialized loss function ($L_{GC}$), the method eliminates common artifacts and enables real-time, user-controlled texture generation.

Introduction

The challenge of texture synthesis lies in maintaining both local detail and global structure. Traditional methods based on Markov Random Fields (MRFs) typically "copy-paste" patches, which preserves sharpness but can fail on complex semantic structures. Modern neural methods, such as CNNMRF, often result in blurry outputs due to the "averaging" effect of Euclidean distance optimization in feature space. Furthermore, these methods frequently suffer from unnatural repetition when the algorithm over-uses a single "best-fit" source patch.

Guided Correspondence Framework

The proposed framework addresses these limitations through two primary innovations: the Guided Correspondence Distance (GCD) and the Guided Correspondence Loss ($L_{GC}$).

Guided Correspondence Distance (GCD)

To prevent the repetitive patterns common in neural synthesis, the authors introduce an occurrence penalty. This mechanism tracks the frequency of source patch usage; as a patch is used more often, its perceived distance increases, forcing the selection of diverse alternatives. Additionally, guidance channels allow users to incorporate spatial constraints, such as orientation fields or semantic masks, directly into the patch-matching process.

Guided Correspondence Loss ($L_{GC}$)

The core of the method's ability to produce sharp textures is the $L_{GC}$. Unlike standard $L_2$ losses that minimize the average distance to all similar patches, $L_{GC}$ employs a "winner-takes-all" approach inspired by contextual loss. It requires a target patch to be significantly closer to its single best-match source patch than to any others. This prevents the optimization from "hedging its bets" and producing blurry averages.

Overview of the Guided Correspondence framework showing optimization-based results (left) and real-time synthesis with user control over orientation and progression (right).

Implementation and Results

The framework is versatile, supporting both high-quality iterative optimization and real-time synthesis when used as a loss function to train Generative Adversarial Networks (GANs). The system utilizes a pre-trained VGG-19 network as a feature extractor to calculate correspondence distances.

Performance and Quality

Evaluated on a dataset of 50 diverse textures, the method consistently outperforms prior state-of-the-art techniques like SinGAN and CNNMRF. Qualitative results show significantly sharper details and better structural coherence. In a user study involving 40 participants and 2,000 questions, the proposed method was preferred by a large margin over other neural approaches.

User Control

One of the standout features is the degree of interactive control. Users can guide the synthesis process using orientation maps (e.g., for wood grain) or progression maps (e.g., for growing biological textures). While optimization-based synthesis offers the highest fidelity (taking 5–20 minutes), the GAN-based approach allows for these edits in real-time via a single forward pass.

Conclusion

By rethinking patch correspondence through the lens of diversity and competitive similarity, this work successfully bridges the gap between classical patch-based synthesis and modern neural networks. The Guided Correspondence framework provides a robust tool for high-quality, controllable texture generation suitable for both offline rendering and interactive design.

Summary updated

Abstract

Markov random fields (MRFs) are the cornerstone of classical approaches to example-based texture synthesis. Yet, it is not fully valued in the deep learning era. This pa-per aims to re-promote the combination of MRFs and neural networks, i.e., the CNNMRF model, for texture synthesis, with two key observations made. We first propose to compute the Guided Correspondence Distance in the nearest neighbor search, based on which a Guided Correspondence loss is defined to measure the similarity of the output texture to the example. Experiments show that our approach sur-passes existing neural approaches in uncontrolled and con-trolled texture synthesis. More importantly, the Guided Cor-respondence loss can function as a general textural loss in, e.g., training generative networks for real-time controlled synthesis and inversion-based single-image editing. In con-trast, existing textural losses, such as the Sliced Wasserstein loss, cannot work on these challenging tasks.
BibTeX
@inproceedings{zhou2023neural,
  author = {Yang Zhou and Kaijian Chen and Rongjun Xiao and Hui Huang},
  title = {{Neural Texture Synthesis with Guided Correspondence}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01735},
}
Figure