TexPainter: Generative Mesh Texturing with Multi-view Consistency

Hongkun Zhang, Zherong Pan, Congyi Zhang, Lifeng Zhu, Xifeng Gao

TL;DR

This paper introduces TexPainter, a novel method for generating high-quality, multi-view consistent textures for 3D models using pre-trained diffusion models. By optimizing latent codes in the color space and avoiding sequential view dependencies, TexPainter significantly improves texture quality and consistency compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

The advent of powerful pre-trained diffusion models has opened doors for automatic texture generation for 3D models. However, a major hurdle is ensuring multi-view consistency – that the generated texture looks coherent when rendered from different camera angles. Existing methods often struggle with this, either by processing views sequentially (leading to suboptimal results) or by manipulating latent codes in a way that degrades image quality. These approaches also tend to make assumptions about inter-view correlations that are not always valid. TexPainter aims to bridge this gap by providing a method that enforces multi-view consistency without these drawbacks.

Methods

TexPainter builds upon the Denoising Diffusion Implicit Models (DDIM) scheme. The core innovation lies in how it enforces multi-view consistency:

  • Color-Space Fusion and Optimization: Instead of directly manipulating noisy latent codes, TexPainter decodes predicted latent codes for each view into the color space. These color images are then fused into a single, common texture image using a weighted averaging scheme that prioritizes pixels nearly orthogonal to the surface. Crucially, an optimization step is then applied to adjust the noiseless latent codes (predicted by DDIM) for each view. This optimization ensures that when these adjusted latents are decoded, they produce images that match the fused color texture when rendered from their respective camera views. This effectively enforces consistency in the color space.
  • Relaxed Sequential Dependency: Unlike prior methods that assume a sequential relationship between views (e.g., view N+1 refines view N), TexPainter treats all views in parallel. The optimization step ensures consistency across all views simultaneously.
  • DDIM for Noiseless Latents: The method leverages DDIM's ability to predict a noiseless latent code at the 0th timestep. This noiseless latent is more amenable to optimization for consistency compared to noisy latents.
  • Extension for Joint Optimization: An optional extension allows for a joint optimization of both the latent codes and the fused color texture. This can further improve quality, especially in areas less well-sampled by camera views, but comes at a higher computational cost.

Datasets

The paper evaluates its method on a collection of 64 high-quality 3D models sourced from various datasets and online repositories. They also use 182 pairs of meshes and text prompts for experiments. For quantitative evaluation, they use images generated by Stable-Diffusion-2-Depth (SD2-depth) as a ground truth dataset, rendering their generated textures from 8 camera views and comparing them to SD2-depth outputs.

Key Insights and Ideas

  • Color Space is Key for Consistency: The paper's central insight is that enforcing consistency in the color space, rather than the latent space, is more effective. Latent codes can be highly non-linear, meaning consistency in latent space doesn't guarantee visual consistency in color.
  • Noiseless Latents are Easier to Align: DDIM's prediction of a noiseless latent code provides a stable target for optimization, making it easier to achieve multi-view consistency compared to working with noisy latents from DDPM.
  • Parallel Processing is More Robust: Abandoning sequential view dependencies simplifies the problem and avoids the artifacts that arise from such assumptions.
  • Weighted Averaging for Fusion: The simple yet effective weighted averaging scheme for fusing color textures helps to prioritize relevant views and pixels.

Key Results

  • Improved Texture Quality: TexPainter consistently generates higher quality textures with better multi-view consistency compared to state-of-the-art methods like TEXTure, TexFusion, Meshy, and Fantasia3D.
  • Quantitative Improvement: FID scores show TexPainter outperforming other methods, indicating better visual quality and similarity to ground truth.
  • Efficiency: The proposed method achieves competitive inference times, with the core algorithm taking around 25-30 minutes per texture on their hardware. The direct encoding extension further reduces this time.
  • Qualitative Examples: Figures showcase impressive results on diverse models (e.g., Albert Einstein bust, cameras, helmets, cars) with various text prompts, demonstrating the method's versatility.

Limitations and Future Work

  • Fixed Camera Views: The current method uses a fixed set of camera views. The authors suggest that selecting dynamic views could improve coverage but acknowledge the non-trivial challenge of integrating this with their parallel processing approach.
  • Inherited LDM Limitations: TexPainter's texture quality is still limited by the underlying pre-trained LDM, meaning sharp features or specific directional constraints might not be perfectly captured.
  • Potential for Blurriness: Inadequate optimization or weighted averaging can sometimes lead to blurry regions or smooth edges.
  • Computational Cost: While efficient, the repeated optimization step per denoising step increases computational cost compared to some prior methods.
  • Future Directions: The authors suggest exploring better pre-trained models, texture up-sampling techniques, fine-tuning diffusion models, and developing lossless encoders to improve performance and quality. They also envision extending the framework to handle multiple sets of prompts or other complex constraints between diffusion processes.

Summary updated

Abstract

The recent success of pre-trained diffusion models unlocks the possibility of the automatic generation of textures for arbitrary 3D meshes in the wild. However, these models are trained in the screen space, while converting them to a multi-view consistent texture image poses a major obstacle to the output quality. In this paper, we propose a novel method to enforce multi-view consistency. Our method is based on the observation that latent space in a pre-trained diffusion model is noised separately for each camera view, making it difficult to achieve multi-view consistency by directly manipulating the latent codes. Based on the celebrated Denoising Diffusion Implicit Models (DDIM) scheme, we propose to use an optimization-based color-fusion to enforce consistency and indirectly modify the latent codes by gradient back-propagation. Our method further relaxes the sequential dependency assumption among the camera views. By evaluating on a series of general 3D models, we find our simple approach improves consistency and overall quality of the generated textures as compared to competing state-of-the-arts. Our implementation is available at: https://github.com/Quantuman134/TexPainter
BibTeX
@inproceedings{zhang2024texpainter,
  author = {Hongkun Zhang and Zherong Pan and Congyi Zhang and Lifeng Zhu and Xifeng Gao},
  title = {{TexPainter: Generative Mesh Texturing with Multi-view Consistency}},
  booktitle = {Siggraph (conference Paper Track)},
  year = {2024},
  doi = {10.1145/3641519.3657494},
}

Loading related content...

Figure