Abstract
LumiTex is an end-to-end framework designed to generate high-fidelity Physically-Based Rendering (PBR) textures for 3D meshes from simple image prompts. By explicitly modeling "illumination context" through a multi-branch transformer architecture and utilizing a geometry-guided view synthesis model for seamless inpainting, it effectively disentangles material properties from lighting. LumiTex significantly outperforms both open-source and commercial state-of-the-art methods in texture quality and relighting accuracy.
Introduction
In modern computer graphics, PBR textures—comprising albedo, metallic, and roughness maps—are essential for realistic rendering as they define how surfaces interact with light. However, generating these from a single reference image is a classic "ill-posed" problem. A primary challenge is the entanglement of material and lighting: models often "bake" shadows or highlights from the reference image into the albedo map, leading to visual inconsistencies when the asset is placed in new environments. Furthermore, traditional texture inpainting methods often struggle to maintain semantic or geometric consistency across occluded regions of a 3D mesh.
Methodology
LumiTex introduces a three-pronged approach to solve the challenges of material decomposition and texture consistency:
Multi-Branch Generation Scheme
Instead of predicting all texture maps in a single pass, LumiTex utilizes a multi-branch Diffusion Transformer (DiT) based on the FLUX.1-dev architecture. The model is trained using flow matching, which provides a stable path for generative modeling. One branch specifically reconstructs "shaded images" to capture the illumination context of the reference object.
Lighting-Aware Material Attention
The core innovation of LumiTex is the cross-attention mechanism between branches. The albedo and metallic-roughness (MR) branches "query" the illumination context from the shaded branch. This allows the model to reason about the location of highlights and shadows, enabling it to strip these effects from the albedo map and correctly assign them to the MR maps.
Geometry-Guided LVSM Inpainting
To handle occluded regions, LumiTex avoids traditional UV-space inpainting, which often results in seams. Instead, it employs a Large View Synthesis Model (LVSM) conditioned on the mesh geometry to hallucinate consistent viewpoints. These views are projected back onto the mesh to create a seamless, 360-degree texture.
Datasets and Training
The authors curated a massive dataset of 92,000 high-quality 3D objects from Objaverse and Objaverse-XL. For each object, 30 views were rendered under three distinct environment maps. This provided high-resolution (1024x1024) ground truth maps for albedo, metallic, roughness, and HDR shaded images, facilitating the supervision required for the multi-branch architecture.
Key Results
LumiTex achieves significant improvements over baselines such as Hunyuan3D-2.1 and DreamMat:
- Quantitative Performance: LumiTex achieved a FID score of 160.8 for texture evaluation, substantially leading the next best open-source model (Hunyuan3D-2.1 at 196.6).
- Relighting Quality: In relighting tests, LumiTex demonstrated much lower error rates, proving successful decoupling of material from light.
- User Preference: Professional 3D modelers rated LumiTex significantly higher (4.48/5.0) than competitors across rendering quality, albedo accuracy, and texture completeness.
Conclusion and Future Work
LumiTex represents a major advancement in high-fidelity texture generation by leveraging illumination context. While highly effective, the authors note that the model's performance is currently dependent on the quality of the input mesh and does not yet explicitly handle complex optical properties like translucency or clear-coat layers. Future work will focus on expanding the model's capability to handle a wider variety of complex materials and improving computational efficiency.
Summary updated