UV-free Texture Generation with Denoising and Geodesic Heat Diffusions

Simone Foti, Stefanos Zafeiriou, Tolga Birdal

TL;DR

UV3-TeD introduces a novel paradigm for texturing 3D assets by generating textures directly on mesh surfaces as colored point clouds, completely bypassing traditional UV-unwrapping. By combining denoising diffusion probabilistic models (DDPMs) with surface-constrained geodesic heat diffusion and a novel farthest-sampled attention mechanism, the method produces high-quality, seam-free, and multi-view consistent textures across diverse object categories.

What Problem Does This Paper Address?

In 3D graphics, texturing a mesh traditionally requires "UV-unwrapping"—flattening the 3D surface into a 2D plane so a standard 2D image texture can be wrapped around it. This process is notoriously problematic. Automatic unwrapping algorithms frequently introduce geometric distortions, visible seams where the texture meets, wasted texture space, and inconsistent resolution across the object's surface.

While recent deep learning methods try to automate or improve this pipeline, most still operate in 2D UV-space or project 2D images onto 3D shapes, inheriting these classic artifacts. The authors ask a fundamental question: What if we could directly generate textures on the curved surface of the object itself?

Methods

The authors propose UV3-TeD (UV-free Texture Diffusion), a generative framework that represents textures as unstructured, colored point clouds sampled directly on a mesh's surface.

The core architecture is a U-Net-style denoising diffusion model built out of Attention-enhanced Heat Diffusion Blocks. It relies on several key innovations:

  • Geodesic Heat Diffusion: Instead of using standard Euclidean distance (which ignores the shape's topology), the model uses heat diffusion over the surface to let points communicate with their geometric neighbors.
  • Farthest-Sampled Attention: Standard self-attention across thousands of texture points is computationally prohibitive. To solve this, the model diffuses features across the surface, extracts a sparse subset of points using Farthest Point Sampling (FPS), performs multi-head self-attention on this compact subset, and diffuses the results back out to the full point cloud.
  • Mixed Robust Laplacian ($L^R_{mix}$): Real-world 3D models often contain topological errors or disconnected components (e.g., a chair leg not perfectly welded to the seat). Standard mesh Laplacians cause heat diffusion to get "stuck." The authors introduce a mixed operator that blends a robust mesh Laplacian with a point-cloud Laplacian, allowing information to bridge small topological gaps while still respecting the underlying geometry.
The UV3-TeD framework separates heavy geometric precomputation from fast online sampling during training.
  • Online Sampling Strategy: To keep training fast, the model uses Poisson Disk Sampling (PDS) to dynamically sample point clouds and interpolate precomputed mesh eigenvectors on the fly, avoiding costly recomputations.

Datasets

The model was trained and evaluated on two primary datasets:

  1. ShapeNet (Chairs category): A standard benchmark for 3D texture generation containing 4,633 training shapes.
  2. Amazon Berkeley Objects (ABO): A challenging, multi-class dataset containing 6,476 shapes across various real-world product categories (sofas, lamps, tables, etc.).
  3. CelebA (Projected): Used in a specialized experiment where 2D face images were projected onto a 3D rippled plane to test the model's ability to handle high-frequency details.

Key Insights and Ideas

  • Attention via Diffusion: The "aha moment" for long-range consistency is using heat diffusion as a downsampling and upsampling mechanism for attention. By diffusing features before sampling them, the sparse points carry aggregate local context, making the subsequent self-attention layer incredibly effective at ensuring global texture harmony.
  • The Mixed Laplacian Compromise: Pure mesh-based Laplacians fail on broken geometry, while pure point-cloud Laplacians ignore topology entirely (causing texture to bleed through thin structures, like from the back of a chair to the front). Blending them via $L^R_{mix}$ provides the perfect balance for real-world, imperfect meshes.
Comparison of heat diffusion: Mesh LBO gets stuck on disconnected parts, Point Cloud LBO bleeds through thin structures, while the proposed Mixed LBO correctly follows the surface.

Key Results

  • Quantitative Superiority: On the ShapeNet chairs dataset, UV3-TeD significantly outperformed the state-of-the-art Point-UV Diffusion baseline, achieving a Frechet Inception Distance (FID) of 54.20 (compared to 63.35 for the baseline) and a Kernel Inception Distance (KID $\times 10^{-4}$) of 42.17 (compared to 83.19).
  • Ablation Validation: Removing the farthest-sampled attention layer caused the most drastic drop in performance (FID degraded from 77.14 to 83.16 on the ABO dataset), proving its necessity for realistic, uniform textures.
  • Massive Speedup: The authors' custom batched tensor implementation for spatial gradient computation achieved a 35× to 38× speedup over traditional nested-loop implementations, enabling viable online training.

Limitations and Future Work

While UV3-TeD represents a major leap toward UV-free workflows, some limitations remain:

  • High-Frequency Details: The model struggles to generate extremely fine, high-frequency textures (like intricate wood grains or fabric weaves) due to the practical limits of point cloud sampling resolutions.
  • Generation Speed: Generating a single texture takes roughly 1 minute and 30 seconds, which is relatively slow for real-time applications.
  • Albedo Only: Currently, the model only generates diffuse albedo colors. The authors note that extending the framework to generate full Bidirectional Reflectance Distribution Functions (BRDFs)—including roughness and metallicity—is a crucial next step for true photorealism.

Summary updated

Abstract

Seams, distortions, wasted UV space, vertex-duplication, and varying resolution over the surface are the most prominent issues of the standard UV-based texturing of meshes. These issues are particularly acute when automatic UV-unwrapping techniques are used. For this reason, instead of generating textures in automatically generated UV-planes like most state-of-the-art methods, we propose to represent textures as coloured point-clouds whose colours are generated by a denoising diffusion probabilistic model constrained to operate on the surface of 3D objects. Our sampling and resolution agnostic generative model heavily relies on heat diffusion over the surface of the meshes for spatial communication between points. To enable processing of arbitrarily sampled point-cloud textures and ensure long-distance texture consistency we introduce a fast re-sampling of the mesh spectral properties used during the heat diffusion and introduce a novel heat-diffusion-based self-attention mechanism. Our code and pre-trained models are available at github.com/simofoti/UV3-TeD.
BibTeX
@inproceedings{foti2024uv,
  author = {Simone Foti and Stefanos Zafeiriou and Tolga Birdal},
  title = {{UV-free Texture Generation with Denoising and Geodesic Heat Diffusions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Cb1Md0RvqF},
}
Figure