MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesis

Di Luo, Shuhui Yang, Mingxin Yang, Jiawei Lu, Yixuan Tang, Xintong Han, Zhuo Chen, Beibei Wang, Chunchao Guo

Physically-based rendering (PBR) materials—comprising maps such as basecolor, roughness, and normals—are the essential building blocks of photorealistic 3D graphics. Despite their importance, the creation of high-quality PBR materials remains a significant bottleneck in digital content creation, typically requiring intensive manual labor or specialized scanning hardware.

MatPedia introduces a generative foundation model that unifies material generation and decomposition into a single architecture. By employing a novel joint RGB-PBR representation and treating material maps as a temporal sequence, MatPedia achieves high-fidelity, native 1024×1024 synthesis that significantly outperforms existing task-specific pipelines.

Addressing Fragmentation and Data Scarcity

Existing AI approaches to material synthesis often suffer from two primary limitations. First is task fragmentation: current models are typically specialized for a single function, such as text-to-material generation or intrinsic decomposition (extracting materials from a single photograph). Second is data scarcity: high-quality PBR datasets are orders of magnitude smaller than standard RGB image datasets. This lack of data makes it difficult for models to bridge the gap between visual appearance and physical properties, often leading to low-resolution outputs or artifacts like "baked-in" lighting.

The MatPedia Framework: Material as Video

The core innovation of MatPedia is the joint RGB-PBR representation. The researchers conceptualized the relationship between an RGB image and its corresponding PBR maps (Basecolor, Normal, Roughness, and Metallic) as being analogous to consecutive frames in a video. Just as video frames share temporal coherence, these five components are physically coupled and share structural details.

The MatPedia architecture, featuring a 3D Causal VAE and a Video Diffusion Transformer (DiT) adapted for material synthesis tasks.

Architectural Components

  1. 3D Causal VAE: MatPedia adapts a spatio-temporal Variational Autoencoder (VAE) to compress the 5-frame sequence into a compact latent space. Crucially, the model uses asymmetric encoding: PBR maps are treated as incremental refinements conditioned on the RGB appearance. This avoids redundant encoding of structural details and ensures the physical maps remain aligned with the visual appearance.
  2. Video Diffusion Transformer (DiT): The model utilizes a pre-trained video DiT as its backbone. Through Low-Rank Adaptation (LoRA), the model is fine-tuned to handle three distinct tasks within a single framework:
  3. Text-to-Material: Generating a full PBR set from descriptive prompts.
  4. Image-to-Material: Rectifying perspective-heavy or distorted photos into flat, usable maps.
  5. Intrinsic Decomposition: Disentangling physical properties from standard planar images.

Data Strategy: MatHybrid-410K

To overcome data scarcity, the authors introduced MatHybrid-410K, a massive mixed corpus. This dataset includes an RGB Appearance Subset of approximately 50,000 planar material images (including real-world photos and procedurally generated images via Gemini 2.5) paired with rich text descriptions. This is complemented by a Complete PBR Subset of 6,000 high-quality PBR sets, rendered into hundreds of thousands of pairs to train decomposition and rectification tasks. This hybrid approach allows the model to inherit a vast "visual vocabulary" from natural images while maintaining physical accuracy.

Key Results and Performance

MatPedia represents a significant leap in material synthesis quality. While previous models were often limited to 256 or 512 resolutions, MatPedia generates native 1024×1024 maps, which are further upsamplable to 4K.

Qualitative comparison of MatPedia against existing methods across various material types and PBR maps.

Quantitatively, MatPedia outperforms state-of-the-art models like MatFuse in both semantic accuracy (CLIP score) and visual quality (DINO-FID). In decomposition tasks, it demonstrates superior robustness in disentangling shadows and highlights from the actual basecolor. Furthermore, the system is highly efficient, capable of generating a complete set of high-resolution PBR maps in approximately 20 seconds.

Limitations and Future Outlook

While MatPedia sets a new standard, some challenges remain. The coupling of spatial features in the 3D VAE makes traditional "noise rolling" for seamless tiling difficult to implement directly. Additionally, the current model focuses on the four primary maps (Basecolor, Normal, Roughness, Metallic); future iterations could expand to include height, opacity, or subsurface scattering. Despite these areas for growth, MatPedia proves that treating materials as a unified physical sequence is a powerful path toward universal material foundation models.

Summary updated

BibTeX
@inproceedings{luo2026matpedia,
  author = {Di Luo and Shuhui Yang and Mingxin Yang and Jiawei Lu and Yixuan Tang and Xintong Han and Zhuo Chen and Beibei Wang and Chunchao Guo},
  title = {{MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=lXUKxx4AGZ},
}
Figure