TL;DR
Current generative models for 3D materials are bottlenecked by simple, single-layer training data that fails to capture real-world complexity. This paper introduces a procedural pipeline that "uplifts" basic PBR materials into visually rich, multi-layered neural materials (capturing effects like dust, clearcoat, and fuzz) and demonstrates how to use this enhanced data to train advanced text-to-material diffusion models.
What Problem Does This Paper Address?
Generative AI has made massive strides in creating 3D shapes, but texturing those shapes realistically remains a challenge. Most large-scale datasets rely on simplified Physically Based Rendering (PBR) models consisting of a basic diffuse component and a single GGX specular lobe (roughness and metalness).
Real-world surfaces, however, are rarely this simple. They feature complex, multi-layered optical effects: the deep gloss of a car's clearcoat, a fine layer of peach fuzz on a piece of fruit, or accumulated dust on an old chair. Manually authoring these high-fidelity materials at scale is prohibitively expensive. This paper tackles this data scarcity problem by procedurally upgrading simple, widely available PBR textures into highly expressive, multi-layered material representations.
Methods
The authors present an end-to-end pipeline that moves from basic textures to generative neural materials:
- Procedural Uplifting: The standard single-lobe specular model is expanded into a generalized non-diffuse reflection model containing eight distinct lobes. This includes a split specular component (a sharp "core" lobe and a blurry "haze" lobe to create realistic double-highlights) and optional "decorator" lobes for dust, fuzz, clearcoat, and internal scattering.
- Neural Material Compression: Managing 22 analytical parameters per material is too heavy for generative models. The authors train a universal MLP (encoder-decoder) to compress these parameters into a compact 6D latent space (represented as two RGB latent textures), while keeping the standard Lambertian diffuse component separate.
- Latent Space Regularization: Autoencoders often create "brittle" latent spaces where slight perturbations cause rendering artifacts (like
Infvalues). To smooth the space for generative models, the authors inject uniform noise ($\pm0.005$) during training and introduce a "random infill" class of physically implausible but mathematically valid materials. - Generative Diffusion: To prove the utility of the dataset, they finetune a video diffusion model (NVIDIA's Cosmos) to predict these 6D neural latent textures directly from text prompts and geometric guides (G-buffers), baking them back onto 3D meshes.
Datasets
- Neural Material Training: The universal MLP was trained on a procedurally generated dataset of millions of materials divided into five classes (Haze, Dust, Clearcoat, Scattering, and Random Infill).
- 3D Asset Dataset: The authors applied their uplifting pipeline to 10,640 3D models sourced from Objaverse and BlenderVault, creating a massive new benchmark of expressive neural materials on actual 3D shapes.
Key Insights and Ideas
- The Power of Double Highlights: Realism often lives in multi-scale roughness. Splitting a single specular lobe into a correlated "core" and "haze" pair naturally replicates how real surfaces catch light at different scales.
- "Trash" Data Stabilizes Generative Spaces: Purely physical priors leave empty gaps in a neural network's latent space. Dedicating 20% of the training data to completely random, physically unrealistic parameter combinations ("infill") prevents the downstream diffusion model from generating broken, unrenderable pixels.
- Decoupled Diffuse Components: By keeping the Lambertian diffuse term separate from the neural non-diffuse layers, the representation remains backward-compatible with standard rendering utilities like albedo-based denoisers.
Key Results
- Quantifiable Realism: When evaluated against a state-of-the-art baseline (TRELLIS.2), the proposed method showed substantial improvements across both perceptual and reconstruction metrics. The CLIP-FID score dropped significantly from 8.642 (TRELLIS.2) to 3.907 (using reference base colors), indicating much higher visual fidelity.
- Reconstruction Accuracy: The optimized 6D latent textures successfully compressed the 22-parameter analytical model with minimal error, maintaining a high peak signal-to-noise ratio (PSNR of 32.53 dB).
- Prompt-Driven Editing: The finetuned diffusion model successfully generated distinct material variations (e.g., adding dust or clearcoat layers) on the same 3D asset based purely on text prompt modifications.
Limitations and Future Work
- Scale-Agnostic Masking: The procedural masks used to simulate dust accumulation or clearcoat scratches do not automatically adapt to the physical size of a 3D mesh, occasionally causing frequency mismatches.
- Diffusion Resolution Limits: While the neural material representation can support high-frequency details, the video diffusion generation stage can sometimes smooth out fine details (like subtle dust variations) due to current resolution constraints in video models.
- Lobe Scalability: Adding even more complex phenomena (like iridescence or thin-film interference) will require expanding the latent space dimensionality, which complicates network regularization.
Summary updated