UltraTex: Aligning Physical Knowledge for High-Fidelity PBR Texture Generation

Jingzhi Bao, Xiaohe Ma, Hong Li, Yuanming Hu, Xiaoguang HanarXiv 2026 · DOI · arXiv

TL;DR

UltraTex is a new cascaded multi-view diffusion framework that generates high-fidelity, physically accurate PBR (Physically Based Rendering) textures from a single reference image. By integrating "world knowledge" from Vision-Language Models (VLMs) and a novel photometric alignment loss, it successfully disentangles material properties from lighting, allowing 3D assets to be realistically relit in any environment.

What Problem Does This Paper Address?

Creating high-quality 3D assets usually requires a professional artist to manually author PBR texture maps—specifically Albedo (base color), Metallic, and Roughness (MR). While recent AI models can generate realistic-looking images, they struggle with "inverse rendering": the ability to look at a single photo and figure out which parts are shiny metal versus matte plastic.

The core challenge is ambiguity. A bright spot in a photo could be a white paint pattern (Albedo), a reflection on polished chrome (Metallic), or a highlight on a wet surface (Roughness). Most current generative methods suffer from "illumination leakage," where shadows and highlights from the original photo are "baked" into the texture, making the 3D object look wrong when placed in a new lighting environment.

Methods

UltraTex uses a two-stage, multi-view diffusion pipeline to solve this ambiguity:

  1. Stage 1: Albedo and Shading Normal Generation. Instead of just predicting color, the model jointly predicts Albedo maps and world-space shading normals. By predicting the "micro-geometry" (the tiny bumps on a surface) alongside the color, the model produces much sharper details and avoids artifacts caused by messy UV mapping.
  2. Stage 2: Metallic-Roughness (MR) Generation. This stage uses two clever tricks to get the physics right:
  3. VLM-Guided Intrinsic Material Adapter: The authors argue that to know if something is metal, the model needs to understand what the object is. They use a lightweight adapter to pull semantic features from the hidden layers of a Vision-Language Model (Qwen3.5-4B). This gives the model "part-level knowledge anchors"—for example, recognizing that a specific part is a "brass gear" and should therefore be metallic.
  4. Photometric Representation Alignment: Since you can't "see" roughness directly, the model is trained to align its internal features with how a material would look under a controlled "flash" light. This helps the model learn the visual signatures of different material-light interactions.
The VLM-guided adapter extracts semantic material priors to guide the MR generation.

Datasets

The researchers curated a massive dataset of 130,000 3D objects sourced from Objaverse-XL and TexVerse. For each object, they generated 32 reference images under random lighting and extracted ground-truth PBR maps (albedo, metallic, roughness) and normals to provide high-quality supervision during training.

Key Insights and Ideas

  • Semantics Matter for Physics: Purely visual patterns aren't enough to distinguish materials. Using a VLM allows the model to perform "material reasoning"—understanding that a sword blade is metal while the leather grip is not, even if they have similar brightness in the source image.
  • Joint Normal Estimation: Predicting shading normals (surface orientation) alongside albedo acts as a powerful regularizer. It forces the model to account for geometry, leading to textures that feel "carved" rather than just "painted" on.
  • Controlled Lighting as a Teacher: By using a "photometric response" (how a material looks under a single point light) as a training target, the model learns to associate specific visual cues—like the sharpness of a highlight—with the underlying roughness value.
The VLM-guided adapter extracts semantic material priors to guide the MR generation.

Key Results

UltraTex significantly outperforms both open-source (e.g., LumiTex, TRELLIS.2) and commercial (e.g., Meshy 6, TripoAI) baselines:

  • Quantitative: It achieved the lowest Mean Squared Error (MSE) for both metallic and roughness attributes and the best scores in perceptual metrics like FID and LPIPS.
  • Qualitative: As seen in the gallery, UltraTex produces textures with zero "baked-in" lighting and highly detailed, spatially varying roughness.
  • User Preference: In a blind study, 31 professional 3D modelers overwhelmingly preferred UltraTex, giving it the highest scores for rendering quality, albedo accuracy, and material plausibility.
Fig. 1. PBR texture gallery generated by UltraTex. Our method produces detailed and physically plausible PBR materials across diverse assets under environmental lighting. In the first row, the right column shows the generated albedo, metallic, and roughness maps from top to bottom. In the second row, a close-up render (right) of the boxed region (left) reveals fine-grained material details. Project page: https://ultratex.vercel.app/.

Limitations and Future Work

While UltraTex is a major step forward, the authors note a few limitations:

  • Standard PBR Only: The model currently focuses on the standard Albedo-Metallic-Roughness stack. It cannot yet faithfully represent complex materials like transparent glass, translucent wax (subsurface scattering), or iridescent "thin-film" coatings.
  • Stylized Assets: On highly stylized or "hand-painted" art, the model sometimes struggles because the original artist may have used non-physical metallic/roughness values for artistic effect, which contradicts the model's "physical knowledge" training.

Future work aims to incorporate more material channels (like transmission and clear coat) to handle a wider variety of real-world and fantasy materials.

Summary updated

Abstract

Generating physically based rendering (PBR) texture maps for a 3D mesh from a single reference image is highly ambiguous, as the observed appearance entangles material identity, surface reflectance, and illumination. While current generative methods produce realistic albedo maps, they still struggle to infer accurate metallic and roughness attributes for relighting. We present UltraTex, a cascaded multi-view framework that moves beyond appearance matching by integrating physical knowledge into multi-view diffusion-based PBR generation. UltraTex first jointly predicts albedo and world-space shading normals to generate more detailed albedo maps, and then predicts metallic-roughness (MR) maps conditioned on the generated albedo, reference image, and material semantic priors. To inject material semantics, a lightweight adapter transforms hidden-layer features of a vision-language model (VLM) into intrinsic material representation tokens, providing part-level knowledge anchors for MR estimation. To model material-light interaction, photometric representation alignment guides intermediate diffusion features toward ground-truth material renderings under controlled lighting. This makes metallic and roughness variations observable through highlight intensity, specular sharpness, and local reflectance contrast, providing stronger guidance for fine-grained material prediction without explicit lighting supervision. Extensive experiments show that UltraTex generates high-quality textures and outperforms existing open-source and commercial methods in visual fidelity, material accuracy, and physical plausibility.
BibTeX
@misc{bao2026ultratex,
  author = {Jingzhi Bao and Xiaohe Ma and Hong Li and Yuanming Hu and Xiaoguang Han},
  title = {{UltraTex: Aligning Physical Knowledge for High-Fidelity PBR Texture Generation}},
  year = {2026},
  eprint = {2611.24941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24941},
}
Figure