GramGAN: Deep 3D Texture Synthesis From 2D Exemplars

Tiziano Portenier, Siavash Arjomand Bigdeli, Orcun GokselNeurIPS 2020 · OpenReview

TL;DR

GramGAN is a novel framework that generates infinite, high-quality 3D textures from a single 2D exemplar image. By using a memory-efficient Multi-Layer Perceptron (MLP) and a unique loss function that combines Generative Adversarial Networks (GANs) with style-transfer concepts, it produces more realistic and diverse 3D materials than previous state-of-the-art methods.

What Problem Does This Paper Address?

Creating realistic 3D textures is a major bottleneck in computer graphics. Traditional photogrammetry (using photos of real objects) is labor-intensive and often results in visible "seams" or distortions when wrapped around complex 3D shapes. Procedural texturing (using algorithms like Perlin noise) avoids these mapping issues but is notoriously difficult to tune to match a specific real-world material.

Recent deep learning approaches using Convolutional Neural Networks (CNNs) produce great 2D results, but they struggle with 3D generation due to massive memory requirements and a lack of "stochasticity"—the natural, non-repeating randomness found in real textures. The authors aim to bridge this gap by creating a system that is memory-efficient, generates infinite 3D volumes, and faithfully matches the "style" of a provided 2D photo.

Methods

The authors introduce GramGAN, which departs from standard 3D CNNs by using a "point operation" model.

GramGAN Architecture
  1. MLP-based Sampler: Instead of generating a whole 3D grid at once, the model uses an MLP that takes a 3D coordinate and noise frequencies as input and outputs a single RGB value. This makes it infinitely scalable and memory-efficient.
  2. Frequency Binning & Injection: Inspired by Perlin noise, the model uses multiple "octaves" of noise. Rather than just feeding noise into the first layer, GramGAN "injects" specific frequency bands into different hidden layers of the MLP. This forces the network to specialize and prevents it from ignoring subtle texture details.
  3. The GramGAN Loss: This is the core innovation. Standard GANs use a discriminator to tell "real" from "fake." GramGAN goes further by calculating Gram matrices (which capture feature correlations, a staple of style transfer) from the discriminator’s own internal layers. By matching these matrices between the exemplar and the generated texture, the model learns the specific "textureness" of the material more effectively than using a generic pre-trained network like VGG.
  4. Random Slicing: During training, the model generates 2D slices of the 3D volume at random orientations to compare against the 2D exemplar. This ensures the 3D texture looks good from any angle, not just along the X, Y, or Z axes.

Datasets

  • Single Exemplars: High-resolution photos of wood and granite were used for ablation studies and direct comparison.
  • Stone Dataset: A collection of 100 stone texture exemplars gathered online to train a "conditional" model capable of navigating a latent texture space.
  • Evaluation Metrics: The authors used SIFID (Single Image Fréchet Inception Distance) to quantitatively measure the similarity between internal patch statistics of the real and generated images.

Key Insights and Ideas

  • Discriminator as a Specialized Stylist: Most style-transfer models use a fixed, pre-trained VGG network to define "style." GramGAN shows that using the evolving features of the discriminator—which is actively learning to distinguish the specific target texture—results in much higher quality and fewer artifacts (avoiding the common "Deep Dream" look).
  • Architecture Matters for Generalization: By injecting scale and bias into hidden layers (similar to StyleGAN’s AdaIN), the model becomes much better at extrapolation—generating textures that look like an exemplar it has never seen before.
  • Independence vs. Structure: Because the model calculates each pixel independently based on its coordinate and noise, it is inherently "stochastic." It won't repeat the exact same pattern twice, which is essential for natural-looking materials like marble or clouds.

Key Results

2D Exemplars to 3D Textures
  • Quantitative: GramGAN achieved significantly lower SIFID scores than the previous state-of-the-art (Henzler et al.). For example, on wood textures, GramGAN scored 0.14 compared to the baseline's 0.67 (lower is better).
  • Qualitative: Visual results show that GramGAN captures anisotropic structures (like the long grains in wood) much better than previous methods, which often produced blurry or "blobby" 3D results.
  • User Study: In a study with 21 participants, users consistently ranked GramGAN's results as more similar to the original reference and higher in quality than the competing framework.

Limitations and Future Work

  • Repetitive Patterns: The model struggles with highly structured, non-stochastic textures like brick walls. Because it synthesizes pixels independently, it lacks the "global" awareness needed to keep rows of bricks perfectly aligned.
  • Complex Stochasticity: Very complex patterns, such as satellite imagery of cities or forests, remain a challenge.
  • Material Properties: The current version only generates "diffuse albedo" (color). The authors suggest that future work could extend this to generate specular maps (shininess) and normal maps (surface bumps) using differentiable renderers.

Summary updated

Abstract

We present a novel texture synthesis framework, enabling the generation of infinite, high-quality 3D textures given a 2D exemplar image. Inspired by recent advances in natural texture synthesis, we train deep neural models to generate textures by non-linearly combining learned noise frequencies. To achieve a highly realistic output conditioned on an exemplar patch, we propose a novel loss function that combines ideas from both style transfer and generative adversarial networks. In particular, we train the synthesis network to match the Gram matrices of deep features from a discriminator network. In addition, we propose two architectural concepts and an extrapolation strategy that significantly improve generalization performance. In particular, we inject both model input and condition into hidden network layers by learning to scale and bias hidden activations. Quantitative and qualitative evaluations on a diverse set of exemplars motivate our design decisions and show that our system performs superior to previous state of the art. Finally, we conduct a user study that confirms the benefits of our framework.
BibTeX
@inproceedings{portenier2020gramgan,
  author = {Tiziano Portenier and Siavash Arjomand Bigdeli and Orcun Goksel},
  title = {{GramGAN: Deep 3D Texture Synthesis From 2D Exemplars}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=klZalmbEEI7},
}
Figure