LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Context

Jingzhi Bao, Hongze Chen, Lingting Zhu, Chenyu Liu, Runze Zhang, Keyang Luo, Zeyu Hu, Weikai Chen, Yingda Yin, Xin Wang, Zehong Lin, Jun Zhang, Xiaoguang Han

Abstract

LumiTex is an end-to-end framework designed to generate high-fidelity Physically-Based Rendering (PBR) textures for 3D meshes from simple image prompts. By explicitly modeling "illumination context" through a multi-branch transformer architecture and utilizing a geometry-guided view synthesis model for seamless inpainting, it effectively disentangles material properties from lighting. LumiTex significantly outperforms both open-source and commercial state-of-the-art methods in texture quality and relighting accuracy.

Introduction

In modern computer graphics, PBR textures—comprising albedo, metallic, and roughness maps—are essential for realistic rendering as they define how surfaces interact with light. However, generating these from a single reference image is a classic "ill-posed" problem. A primary challenge is the entanglement of material and lighting: models often "bake" shadows or highlights from the reference image into the albedo map, leading to visual inconsistencies when the asset is placed in new environments. Furthermore, traditional texture inpainting methods often struggle to maintain semantic or geometric consistency across occluded regions of a 3D mesh.

Comparison of generation schemes: (a) Two-stage, (b) Multi-Channel, and (c) LumiTex's multi-branch approach with lighting-aware attention.

Methodology

LumiTex introduces a three-pronged approach to solve the challenges of material decomposition and texture consistency:

Multi-Branch Generation Scheme

Instead of predicting all texture maps in a single pass, LumiTex utilizes a multi-branch Diffusion Transformer (DiT) based on the FLUX.1-dev architecture. The model is trained using flow matching, which provides a stable path for generative modeling. One branch specifically reconstructs "shaded images" to capture the illumination context of the reference object.

Lighting-Aware Material Attention

The core innovation of LumiTex is the cross-attention mechanism between branches. The albedo and metallic-roughness (MR) branches "query" the illumination context from the shaded branch. This allows the model to reason about the location of highlights and shadows, enabling it to strip these effects from the albedo map and correctly assign them to the MR maps.

The overall architecture of LumiTex, illustrating the Multi-Modal and Multi-View DiT pipelines for PBR texture synthesis.

Geometry-Guided LVSM Inpainting

To handle occluded regions, LumiTex avoids traditional UV-space inpainting, which often results in seams. Instead, it employs a Large View Synthesis Model (LVSM) conditioned on the mesh geometry to hallucinate consistent viewpoints. These views are projected back onto the mesh to create a seamless, 360-degree texture.

Datasets and Training

The authors curated a massive dataset of 92,000 high-quality 3D objects from Objaverse and Objaverse-XL. For each object, 30 views were rendered under three distinct environment maps. This provided high-resolution (1024x1024) ground truth maps for albedo, metallic, roughness, and HDR shaded images, facilitating the supervision required for the multi-branch architecture.

Key Results

LumiTex achieves significant improvements over baselines such as Hunyuan3D-2.1 and DreamMat:

  • Quantitative Performance: LumiTex achieved a FID score of 160.8 for texture evaluation, substantially leading the next best open-source model (Hunyuan3D-2.1 at 196.6).
  • Relighting Quality: In relighting tests, LumiTex demonstrated much lower error rates, proving successful decoupling of material from light.
  • User Preference: Professional 3D modelers rated LumiTex significantly higher (4.48/5.0) than competitors across rendering quality, albedo accuracy, and texture completeness.

Conclusion and Future Work

LumiTex represents a major advancement in high-fidelity texture generation by leveraging illumination context. While highly effective, the authors note that the model's performance is currently dependent on the quality of the input mesh and does not yet explicitly handle complex optical properties like translucency or clear-coat layers. Future work will focus on expanding the model's capability to handle a wider variety of complex materials and improving computational efficiency.

Summary updated

BibTeX
@inproceedings{bao2026lumitex,
  author = {Jingzhi Bao and Hongze Chen and Lingting Zhu and Chenyu Liu and Runze Zhang and Keyang Luo and Zeyu Hu and Weikai Chen and Yingda Yin and Xin Wang and Zehong Lin and Jun Zhang and Xiaoguang Han},
  title = {{LumiTex: Towards High-Fidelity PBR Texture Generation with Illumination Context}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=CDwG0Bebfo},
}
Figure