EucliDreamer: Fast and High-Quality Texturing for 3D Models with Depth-Conditioned Stable Diffusion

Cindy X. Le, Congrui Hetang, Chendi Lin, Ang Cao, Yihui HearXiv 2024 · DOI · arXiv

Abstract

EucliDreamer is a novel framework designed to automate the labor-intensive process of texturing 3D models using text prompts. By integrating depth-conditioned Stable Diffusion with an efficient multiresolution hash-grid representation, the system overcomes common generative issues such as the "Janus problem" (hallucinated multiple faces) and slow convergence. EucliDreamer produces high-quality, geometrically consistent textures approximately twice as fast as previous state-of-the-art methods like DreamFusion.

Introduction

In 3D content creation, geometry is only half the battle. Texturing—the process of painting a 3D mesh to achieve realism—is a notorious bottleneck that often takes professional artists days to complete. While recent generative AI advances have introduced text-to-3D methods, they frequently suffer from two major issues: visual inconsistency due to a lack of structural understanding and extreme computational inefficiency, often requiring tens of thousands of iterations.

EucliDreamer tackles these challenges by providing a way to generate high-fidelity textures that are "pinned" to the actual geometry of the mesh. This ensures better visual consistency and significantly faster results for practical creative workflows.

Methods

EucliDreamer represents the texture of a 3D mesh as an implicit function, specifically a Multi-Layer Perceptron (MLP) that maps 2D surface coordinates to RGB values. To maintain efficiency, the framework utilizes a multiresolution hash grid for compact representation and rapid inference.

The core innovation lies in the optimization process using Depth-Conditioned Score Distillation Sampling (SDS). During training, a differentiable renderer creates 2D images and depth maps of the mesh from random viewpoints. These depth maps are fed back into the diffusion model as a geometric prior, informing the AI exactly where the surface is located. This feedback loop prevents the model from painting features in empty space or misaligning textures with the underlying geometry.

The EucliDreamer training loop utilizes a differentiable renderer and depth-conditioned SDS to optimize a hash-grid texture representation.

Key Results and Insights

EucliDreamer demonstrates a significant performance leap in both speed and quality. The method converges in approximately 4,300 steps, representing a 2x speedup over the previous benchmark, DreamFusion, which typically requires over 10,000 steps.

Qualitatively, the use of depth as a constraint removes visual ambiguity, eliminating the struggle where the AI misplaces features like windows or doors. In a user study involving 3D professionals, EucliDreamer was consistently ranked superior to existing methods like Text2Tex and CLIPMesh. The system successfully captures complex "texture baking" effects, such as realistic specular highlights on metallic surfaces and fine text on small objects.

Qualitative comparison of EucliDreamer against existing texturing methods across various 3D meshes, demonstrating superior consistency and detail.

Limitations and Future Work

Despite its strengths, EucliDreamer faces challenges with extremely complex geometries containing deep occlusions or internal structures that are difficult to render from external camera angles. Additionally, because the underlying diffusion model is trained on real-world photographs, the generated textures often include "baked-in" shadows and highlights. While visually appealing in static renders, this can conflict with dynamic lighting in game engines. Future research aims to fine-tune diffusion models on "albedo-only" datasets to produce textures more compatible with professional VFX pipelines.

Summary updated

BibTeX
@misc{le2024euclidreamer,
  author = {Cindy X. Le and Congrui Hetang and Chendi Lin and Ang Cao and Yihui He},
  title = {{EucliDreamer: Fast and High-Quality Texturing for 3D Models with Depth-Conditioned Stable Diffusion}},
  year = {2024},
  eprint = {2404.10279},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.10279},
}
Figure