Abstract: Recently, deep generative adversarial networks for image generation have advanced rapidly; yet, only a small amount of research has focused on generative models for irregular structures, particularly meshes. Mesh generation and synthesis remains a fundamental topic in computer graphics. In this work, we propose a novel framework for synthesizing geometric textures. It learns geometric texture statistics from local neighborhoods of a single reference 3D model. It learns deep features on the faces of the input triangulation, which is used to subdivide and generate offsets across multiple scales, without parameterization of the reference or target mesh. Our network displaces mesh vertices in any direction, enabling synthesis of geometric textures that cannot be expressed by a simple 2D displacement map. Learning and synthesizing on local geometric patches enables a genus-oblivious framework, facilitating texture transfer between shapes of different genus.
Introduction
In computer graphics, adding fine-grained geometric detail to a 3D model is traditionally a labor-intensive process. Artists usually rely on "displacement mapping," which uses a 2D image to push vertices along their normal direction. However, this approach suffers from two major flaws. First, it requires parameterization, or "unwrapping" the 3D mesh into a 2D plane (UV mapping), which often creates seams and distortion. Second, it faces geometric limitations, as standard displacement only moves vertices along the normal vector, failing to represent complex 3D structures like overlapping scales or hooked thorns that require tangential shifts.
This paper introduces a generative model that learns 3D texture directly from the mesh surface. By operating on local triangular patches, the method is "genus-oblivious"—it functions regardless of whether the model is a sphere or a complex statue—and is capable of full 3D vertex displacement.
Methodology
The core of the approach is a Hierarchical Generative Adversarial Network (GAN) designed specifically for triangular meshes. The system uses a pyramid of generators that starts with a low-resolution template mesh and iteratively subdivides it. At each level, the generator predicts 3D displacement vectors for the vertices to match the texture statistics of the reference model.
Symmetric Face Convolution
Standard CNNs are incompatible with meshes because the "neighborhood" of a triangle is irregular. To solve this, the authors use a novel Symmetric Face Convolution that examines a face and its three neighbors. To ensure the network is invariant to the mesh's initial indexing, they apply a "max" operation, making the operator symmetric.
Local Feature Extraction
Instead of global coordinates, the network learns from local rotation-and-translation-invariant features, such as edge lengths and relative vertex positions. This allows the learned texture to be applied anywhere on a completely different shape. The model is trained using a WGAN-GP (Wasserstein GAN with Gradient Penalty) on a single reference mesh, learning to distinguish between "real" patches from the reference and "fake" patches generated on the target.
Results and Applications
The framework demonstrates impressive qualitative success in transferring textures from natural objects to arbitrary target meshes. Because the network operates on local patches rather than global structure, it can learn a texture from a genus-0 shape (like a ball) and synthesize it onto a high-genus shape without topological errors.
Key advantages of this method include:
- Beyond the Normal: By allowing $(dx, dy, dz)$ displacements, the network can synthesize "overhangs" and tangential details impossible for traditional 2D maps.
- No UVs Required: Working directly in 3D space on mesh faces bypasses the UV unwrapping process entirely.
- Efficiency: Once trained, the synthesis performs in just a few seconds, significantly faster than state-of-the-art parameterization methods like OptCuts.
Limitations and Future Work
While powerful, the current model assumes textures are stationary and isometric, meaning they look the same regardless of orientation. It struggles with directional textures like wood grain or brick patterns. Additionally, the network is designed for local textures and cannot synthesize large-scale semantic features (e.g., growing a face onto a sphere). Future research may involve learning "directional fields" to allow for anisotropic, flow-based texture synthesis.
Summary updated