Deep geometric texture synthesis

Amir Hertz, Rana Hanocka, Raja Giryes, Daniel Cohen-Or

TL;DR

This paper introduces a deep learning framework for synthesizing and transferring geometric textures between 3D meshes. By learning the local statistics of a single reference model, the system can "grow" complex 3D details—like scales, spikes, or organic bumps—onto any target mesh without requiring manual UV mapping or being restricted by the shape's topology (genus).

What Problem Does This Paper Address?

In computer graphics, adding fine-grained geometric detail to a 3D model is traditionally a labor-intensive process. Artists usually rely on "displacement mapping," which uses a 2D image to push vertices along their normal direction. However, this has two major flaws:

  1. Parameterization: You must "unwrap" the 3D mesh into a 2D plane (UV mapping), which often creates seams and distortion.
  2. Geometric Limitation: Standard displacement only moves vertices "up and down" (along the normal). It cannot represent complex 3D structures that involve "sideways" (tangential) shifts, such as overlapping scales or hooked thorns.

The authors aim to create a generative model that learns 3D texture directly from the mesh surface, making it "genus-oblivious" (it doesn't care if your model is a sphere or a donut) and capable of full 3D vertex displacement.

Methods

The core of the approach is a Hierarchical Generative Adversarial Network (GAN) designed specifically for triangular meshes.

  • Symmetric Face Convolution: Standard CNNs don't work on meshes because the "neighborhood" of a triangle is irregular. The authors use a novel convolution that looks at a face and its three neighbors. To ensure the network doesn't get confused by the order of those neighbors, they apply a "max" operation, making the operator symmetric and invariant to the mesh's initial indexing.
  • Multiscale Architecture: The system uses a pyramid of generators. It starts with a low-resolution "template" mesh and iteratively subdivides it. At each level, the generator predicts 3D displacement vectors for the vertices to match the texture statistics of the reference model at that specific scale.
  • Local Feature Extraction: Instead of global coordinates, the network learns from local rotation-and-translation-invariant features (like edge lengths and relative vertex positions). This allows the learned "texture" to be applied anywhere on a completely different shape.
  • Training: The model is trained using a WGAN-GP (Wasserstein GAN with Gradient Penalty) on a single reference mesh. It learns to distinguish between "real" patches from the reference and "fake" patches generated on the target.

Datasets

Because the model learns from a single reference mesh, it does not require a massive labeled dataset. For their experiments and evaluations, the authors used:

  • Thingi10K: A large-scale repository of 3D-printable models used to source various geometric textures (e.g., the "spiky ball" or "thorny lizard").
  • Custom Models: Various standard graphics meshes (the Stanford Bunny, a giraffe, a cow) were used as targets for texture transfer.

Key Insights and Ideas

  • Genus-Oblivious Synthesis: Because the network operates on local triangular patches rather than the global structure, you can learn a texture from a genus-0 shape (like a ball) and perfectly synthesize it onto a genus-4 shape (like a complex statue) without any topological errors.
  • Beyond the Normal: By allowing $(dx, dy, dz)$ displacements, the network can synthesize "overhangs" and tangential details that are mathematically impossible to represent with traditional 2D displacement maps.
  • No UVs Required: By working directly in 3D space on the mesh faces, the method bypasses the "UV unwrapping" nightmare entirely.
  • Latent Space Control: Since it is a generative model, users can sample different noise vectors to produce variations of the same texture, allowing for "stochastic" organic looks rather than repetitive tiling.

Key Results

  • Qualitative Success: The paper demonstrates impressive results transferring textures from natural objects (like a thorny lizard) to arbitrary animals (like a camel or pig).
  • Efficiency: Compared to state-of-the-art parameterization methods like OptCuts, which can take 10 minutes to compute a mapping, this method performs the synthesis in just a few seconds once trained.
  • Resolution Independence: The scale of the synthesized texture is naturally tied to the resolution of the target mesh; a denser mesh results in finer, more detailed texture synthesis.

Limitations and Future Work

  • Isotropy: The current model assumes textures are "stationary and isometric"—meaning they look the same regardless of orientation. It struggles with directional textures, like wood grain or brick patterns, which follow a specific flow.
  • Large-Scale Structures: The network is excellent at "local" textures but cannot currently synthesize large-scale semantic features (e.g., it can't "grow" a face onto a sphere).
  • Mesh Quality: The method requires a relatively well-behaved (locally uniform) triangulation on the target mesh. If the target mesh has extremely stretched or "sliver" triangles, the synthesis quality degrades, often requiring a pre-processing remesh step.
  • Future Direction: The authors suggest that future work could involve learning "directional fields" to allow for anisotropic (flow-based) texture synthesis.

Summary updated

Abstract

Recently, deep generative adversarial networks for image generation have advanced rapidly; yet, only a small amount of research has focused on generative models for irregular structures, particularly meshes. Nonetheless, mesh generation and synthesis remains a fundamental topic in computer graphics. In this work, we propose a novel framework for synthesizing geometric textures. It learns geometric texture statistics from local neighborhoods (i.e., local triangular patches) of a single reference 3D model. It learns deep features on the faces of the input triangulation, which is used to subdivide and generate offsets across multiple scales, without parameterization of the reference or target mesh. Our network displaces mesh vertices in any direction (i.e., in the normal and tangential direction), enabling synthesis of geometric textures, which cannot be expressed by a simple 2D displacement map. Learning and synthesizing on local geometric patches enables a genus-oblivious framework, facilitating texture transfer between shapes of different genus.
BibTeX
@article{hertz2020deep,
  author = {Amir Hertz and Rana Hanocka and Raja Giryes and Daniel Cohen-Or},
  title = {{Deep geometric texture synthesis}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  url = {https://openreview.net/forum?id=ffWzKA6Bl4},
}

Loading related content...

Figure