Toward Richer Material Generation via Procedural Data Enhancement

Yunchen Yu, Jacob Munkberg, Jon Hasselgren, Chris Cummings, Steve Marschner, Andrea Weidlich

TL;DR

Current generative models for 3D materials are bottlenecked by simple, single-layer training data that fails to capture real-world complexity. This paper introduces a procedural pipeline that "uplifts" basic PBR materials into visually rich, multi-layered neural materials (capturing effects like dust, clearcoat, and fuzz) and demonstrates how to use this enhanced data to train advanced text-to-material diffusion models.

What Problem Does This Paper Address?

Generative AI has made massive strides in creating 3D shapes, but texturing those shapes realistically remains a challenge. Most large-scale datasets rely on simplified Physically Based Rendering (PBR) models consisting of a basic diffuse component and a single GGX specular lobe (roughness and metalness).

Real-world surfaces, however, are rarely this simple. They feature complex, multi-layered optical effects: the deep gloss of a car's clearcoat, a fine layer of peach fuzz on a piece of fruit, or accumulated dust on an old chair. Manually authoring these high-fidelity materials at scale is prohibitively expensive. This paper tackles this data scarcity problem by procedurally upgrading simple, widely available PBR textures into highly expressive, multi-layered material representations.

Methods

The authors present an end-to-end pipeline that moves from basic textures to generative neural materials:

  1. Procedural Uplifting: The standard single-lobe specular model is expanded into a generalized non-diffuse reflection model containing eight distinct lobes. This includes a split specular component (a sharp "core" lobe and a blurry "haze" lobe to create realistic double-highlights) and optional "decorator" lobes for dust, fuzz, clearcoat, and internal scattering.
  2. Neural Material Compression: Managing 22 analytical parameters per material is too heavy for generative models. The authors train a universal MLP (encoder-decoder) to compress these parameters into a compact 6D latent space (represented as two RGB latent textures), while keeping the standard Lambertian diffuse component separate.
Schematic of the enhanced 8-lobe BRDF model
  1. Latent Space Regularization: Autoencoders often create "brittle" latent spaces where slight perturbations cause rendering artifacts (like Inf values). To smooth the space for generative models, the authors inject uniform noise ($\pm0.005$) during training and introduce a "random infill" class of physically implausible but mathematically valid materials.
  2. Generative Diffusion: To prove the utility of the dataset, they finetune a video diffusion model (NVIDIA's Cosmos) to predict these 6D neural latent textures directly from text prompts and geometric guides (G-buffers), baking them back onto 3D meshes.
The six enhancement types compared to simple PBR

Datasets

  • Neural Material Training: The universal MLP was trained on a procedurally generated dataset of millions of materials divided into five classes (Haze, Dust, Clearcoat, Scattering, and Random Infill).
  • 3D Asset Dataset: The authors applied their uplifting pipeline to 10,640 3D models sourced from Objaverse and BlenderVault, creating a massive new benchmark of expressive neural materials on actual 3D shapes.

Key Insights and Ideas

  • The Power of Double Highlights: Realism often lives in multi-scale roughness. Splitting a single specular lobe into a correlated "core" and "haze" pair naturally replicates how real surfaces catch light at different scales.
  • "Trash" Data Stabilizes Generative Spaces: Purely physical priors leave empty gaps in a neural network's latent space. Dedicating 20% of the training data to completely random, physically unrealistic parameter combinations ("infill") prevents the downstream diffusion model from generating broken, unrenderable pixels.
  • Decoupled Diffuse Components: By keeping the Lambertian diffuse term separate from the neural non-diffuse layers, the representation remains backward-compatible with standard rendering utilities like albedo-based denoisers.

Key Results

  • Quantifiable Realism: When evaluated against a state-of-the-art baseline (TRELLIS.2), the proposed method showed substantial improvements across both perceptual and reconstruction metrics. The CLIP-FID score dropped significantly from 8.642 (TRELLIS.2) to 3.907 (using reference base colors), indicating much higher visual fidelity.
  • Reconstruction Accuracy: The optimized 6D latent textures successfully compressed the 22-parameter analytical model with minimal error, maintaining a high peak signal-to-noise ratio (PSNR of 32.53 dB).
  • Prompt-Driven Editing: The finetuned diffusion model successfully generated distinct material variations (e.g., adding dust or clearcoat layers) on the same 3D asset based purely on text prompt modifications.

Limitations and Future Work

  • Scale-Agnostic Masking: The procedural masks used to simulate dust accumulation or clearcoat scratches do not automatically adapt to the physical size of a 3D mesh, occasionally causing frequency mismatches.
  • Diffusion Resolution Limits: While the neural material representation can support high-frequency details, the video diffusion generation stage can sometimes smooth out fine details (like subtle dust variations) due to current resolution constraints in video models.
  • Lobe Scalability: Adding even more complex phenomena (like iridescence or thin-film interference) will require expanding the latent space dimensionality, which complicates network regularization.

Summary updated

Abstract

Generative models for material creation are fundamentally limited by the quality and expressivity of available training data. Simple physically based rendering (PBR) materials, which combine a diffuse term with a single-lobe specular component, are commonly used for training but are insufficient to capture many important visual effects present in real materials. We present a method that enhances such simple PBR materials to more expressive ones, by augmenting the single GGX specular lobe into a layered model that captures a broader range of non-diffuse effects. Starting from a simple material, we procedurally construct a corresponding multi-lobe non-diffuse component guided by physical priors, enabling effects such as dust, clearcoat, and layered scattering. To provide a compact representation for downstream applications, we encode this non-diffuse component as a neural material with a shared 6D latent space, where each material instance is represented by two latent textures and decoded by a pretrained universal MLP. We further regularize the latent space to support material generation. The resulting neural material dataset enables training generative models for richer material creation. To demonstrate this application, we finetune a video diffusion model to produce neural latent textures that encode our multi-lobe material, and present generative results as proof of feasibility. Our procedural data enhancement approach is an important step toward improving expressivity in material generation.
BibTeX
@inproceedings{yu2026toward,
  author = {Yunchen Yu and Jacob Munkberg and Jon Hasselgren and Chris Cummings and Steve Marschner and Andrea Weidlich},
  title = {{Toward Richer Material Generation via Procedural Data Enhancement}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2026},
  doi = {10.1145/3799902.3811210},
}
Figure