Abstract
State-of-the-art image-to-3D generative models often produce assets with "baked-in" lighting and simplified geometry, limiting their utility in professional graphics pipelines. This paper introduces a "plug-and-play" framework that upgrades existing models by replacing flat RGB textures with full Physics-Based Rendering (PBR) materials. By fine-tuning Stable Diffusion for intrinsic property estimation and employing an iterative normal refinement process, the framework significantly enhances geometric detail and enables realistic relighting of generated 3D assets.
The "Plastic" Problem in 3D Generation
Current generative models like CRM, InstantMesh, and Wonder3D typically produce 3D meshes with simple RGB textures. These textures suffer from baked-in lighting, meaning shadows and highlights are fixed to the original image's environment. When these assets are placed in new digital scenes, they fail to react naturally to new light sources, resulting in a flat, "plastic" appearance.
Furthermore, there is often a disconnect between texture and geometry; high-frequency details visible in the input image frequently fail to manifest as actual geometric depth on the mesh. This research aims to bridge this gap by transitioning from simple color textures to PBR materials—specifically albedo, roughness, metalness, and bump maps.
Methodology: A Multi-Stage Enhancement Pipeline
The proposed framework is designed to enhance any base image-to-3D model through a three-step process:
- Albedo and Normal Estimation: The authors fine-tuned Stable Diffusion (v2.1) to function as an image-to-image translation tool. Instead of generating images from text, the model predicts a "clean" albedo map (color without lighting) and a high-detail normal map from a single RGB input.
- PBR Material Generation:
- Albedo: The predicted albedo is processed by a standard 3D generator to create a consistent 3D UV map.
- Metalness & Roughness (M&R): To handle semantic properties (e.g., distinguishing gold from wood), the authors use the Segment-Anything-Model (SAM) to create a 3D semantic mask. A Vision-Language Model (VLM) like Gemini then recommends physically plausible M&R values for each segment.
- Iterative Normal Refinement: To address flawed initial geometry, the authors optimize a learnable bump map (parameterized by an MLP). This process minimizes the difference between the rendered mesh normals and the high-quality normals predicted by the diffusion model, effectively "baking" fine-grained details back into the surface.
Data and Training
The diffusion priors were fine-tuned using two primary datasets:
- HyperSim: A synthetic indoor-scene dataset used for initial training on ground-truth albedo and normal maps.
- Objaverse: To bridge the "color space gap" between indoor scenes and isolated objects, the authors performed further fine-tuning on this massive dataset of 3D assets, ensuring accurate albedo maps for diverse objects.
Key Results and Insights
The framework was evaluated against several base models, including CRM, InstantMesh, TripoSR, and Wonder3D. Qualitative results show significantly sharper details and realistic behavior under changing lighting conditions. In a user study with 60 participants, the "boosted" versions were preferred 83.51% of the time over original outputs.
Crucially, the paper demonstrates that Stable Diffusion’s internal world-knowledge can be repurposed to disentangle lighting from color. Additionally, leveraging VLMs for semantic material assignment proves more robust than trying to predict metalness purely from pixel data. While the base generation takes approximately 25 minutes, the boosting process adds only 5 minutes of overhead, making it a practical post-processing step for 3D artists.
Limitations and Future Work
Despite the improvements, the model faces challenges with monochromatic objects (like a plain white sphere) where the normal refinement lacks geometric anchors. There is also inherent randomness in the diffusion process that can lead to minor color hallucinations. Future work will focus on fully automating the 3D semantic masking process to remove the need for semi-automatic intervention.
Summary updated