DualMat: PBR Material Estimation via Coherent Dual-Path Diffusion

Yifeng Huang, Zhang Chen, Yi Xu, Minh Hoai, Zhong Li

TL;DR

This paper introduces DualMat, a novel diffusion-based framework for estimating Physically Based Rendering (PBR) materials from single images. By employing a unique dual-path architecture and feature distillation, DualMat achieves state-of-the-art accuracy in separating albedo, metallic, and roughness properties, significantly improving realism in 3D graphics and enabling better integration with image-to-3D pipelines.

What Problem Does This Paper Address?

Creating realistic 3D assets for games, VR, and AR requires accurately capturing how surfaces interact with light, which is defined by Physically Based Rendering (PBR) materials. Estimating these material properties (like albedo, metallicness, and roughness) from a single 2D image is a notoriously difficult problem. This is because the appearance of a surface is a complex interplay of its intrinsic material properties, the lighting conditions, and the object's geometry. Traditional methods often struggle with high-frequency details, consistency across different views, and the nuanced interplay between different material properties. Furthermore, recent advances in image-to-3D generation often produce assets with simplified or view-dependent textures, lacking the physically accurate materials needed for realistic relighting and integration into virtual environments. DualMat aims to bridge this gap by providing a robust and accurate PBR material estimation method that can be seamlessly integrated into these 3D pipelines.

Methods

DualMat's core innovation lies in its dual-path diffusion framework. Instead of a single network trying to predict all PBR properties, DualMat uses two specialized paths:

  • Albedo-Optimized Path: This path leverages the rich visual priors from a pre-trained diffusion model (like Stable Diffusion) to excel at generating high-quality albedo maps. It uses an RGB encoder and processes material components somewhat independently.
  • Material-Specialized Path: This path is designed for precise metallic and roughness estimation. It uses a unified encoder that jointly processes all PBR components (albedo, metallic, roughness) to capture their intricate interdependencies within a compact latent space.

To ensure these two paths work harmoniously, feature distillation is employed during training. This technique aligns intermediate representations from both paths, ensuring their predictions are coherent.

For efficiency, DualMat utilizes rectified flow, a technique that learns a continuous transformation between noisy and clean data. This allows for significantly fewer inference steps (as few as 2-4) compared to traditional diffusion models, while maintaining high quality.

To handle real-world scenarios, DualMat incorporates:

  • Patch-based estimation for high-resolution inputs, allowing for detailed predictions without losing global consistency.
  • Cross-view attention for multi-view inputs, ensuring that material properties are consistent across different viewpoints, which is crucial for 3D reconstruction.

Datasets

The primary dataset used for training and evaluation is Objaverse. The authors created a large dataset by rendering 65,000 objects from Objaverse from 16 different viewpoints under 100 different HDR environment maps, resulting in over one million image-PBR pairs. For evaluation, a test set of 654 objects from Objaverse was used, rendered under novel HDR environments. The paper also demonstrates results on real-world images, showcasing the model's generalization capabilities.

Key Insights and Ideas

  • Specialization is Key: Separating the PBR estimation task into an albedo-focused path and a material-property-focused path allows each to excel in its domain, leading to superior overall results compared to a single, monolithic approach.
  • Leveraging Pre-trained Models: Utilizing the visual knowledge embedded in large pre-trained diffusion models (like Stable Diffusion) is highly effective for capturing the complex nuances of albedo, a visually rich property.
  • Jointly Learning Interdependencies: For metallic and roughness, which are highly interdependent and less visually distinct than albedo, a unified encoder that learns their relationships directly is crucial for accurate estimation.
  • Feature Distillation for Coherence: Simply having two paths isn't enough; actively forcing them to learn from each other through feature distillation is essential for ensuring their outputs can be seamlessly combined.
  • Rectified Flow for Efficiency: The adoption of rectified flow significantly reduces inference time, making diffusion-based PBR estimation practical for real-time applications and integration into complex pipelines.

Key Results

  • State-of-the-Art Performance: DualMat achieves state-of-the-art results on both Objaverse and real-world datasets.
  • Albedo Improvement: Up to a 28% improvement in albedo estimation accuracy (e.g., PSNR of 28.6 dB, SSIM of 0.932) compared to previous methods.
  • Metallic/Roughness Improvement: A 39% reduction in metallic and roughness estimation errors (RMSE values of 0.057 and 0.060, respectively).
  • Efficient Inference: Achieves high-quality results with as few as 2-4 sampling steps due to rectified flow.
  • Qualitative Superiority: Visual results demonstrate significantly better detail preservation, cleaner material separation, and improved generalization to diverse real-world objects compared to baselines.
  • Multi-view Consistency: The cross-view attention mechanism effectively ensures coherent material predictions across multiple viewpoints.

Limitations and Future Work

  • Memory Cost: The dual-path architecture, while effective, increases memory requirements.
  • Future Work: The authors suggest exploring more efficient architectures to reduce memory footprint. They also envision language-guided material editing as a promising future direction, allowing users to control material properties through text prompts.

Summary updated

Abstract

We present DualMat, a novel dual-path diffusion framework for estimating Physically Based Rendering (PBR) materials from single images under complex lighting conditions. Our approach operates in two distinct latent spaces: an albedo-optimized path leveraging pretrained visual knowledge through RGB latent space, and a material-specialized path operating in a compact latent space designed for precise metallic and roughness estimation. To ensure coherent predictions between the albedo-optimized and material-specialized paths, we introduce feature distillation during training. We employ rectified flow to enhance efficiency by reducing inference steps while maintaining quality. Our framework extends to high-resolution and multi-view inputs through patch-based estimation and cross-view attention, enabling seamless integration into image-to-3D pipelines. DualMat achieves state-of-the-art performance on both Objaverse and real-world data, significantly outperforming existing methods with up to 28% improvement in albedo estimation and 39% reduction in metallic-roughness prediction errors. Our project can be found at yifehuang97.github.io/DualMatProjPage/.
BibTeX
@misc{huang2025dualmat,
  author = {Yifeng Huang and Zhang Chen and Yi Xu and Minh Hoai and Zhong Li},
  title = {{DualMat: PBR Material Estimation via Coherent Dual-Path Diffusion}},
  year = {2025},
  eprint = {2508.05060},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.05060},
}
Figure