Diffusion Time-step Curriculum for One Image to 3D Generation

Xuanyu Yi, Zike Wu, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, Hanwang Zhang

Abstract

Score distillation sampling (SDS) has been widely adopted to overcome the absence of unseen views in reconstructing 3D objects from a single image. It leverages pretrained 2D diffusion models as teacher to guide the reconstruction of student 3D models. Despite their remarkable success, SDS-based methods often encounter geometric artifacts and texture saturation. We find out the crux is the overlooked indiscriminate treatment of diffusion time-steps during optimization: it unreasonably treats the student-teacher knowledge distillation to be equal at all time-steps and thus entangles coarse-grained and fine-grained modeling. Therefore, we propose the Diffusion Time-step Curriculum one-image-to-3D pipeline (DTC123), which involves both the teacher and student models collaborating with the time-step curriculum in a coarse-to-fine manner. Extensive experiments on NeRF4, RealFusion15, GSO and Level50 benchmark demonstrate that DTC123 can produce multiview consistent, high-quality, and diverse 3D assets.

Introduction

Generating a 3D object from a single image is a "severely ill-posed" problem because a single viewpoint provides insufficient information about the object's hidden surfaces. Current state-of-the-art methods address this using Score Distillation Sampling (SDS), where a pre-trained 2D diffusion model acts as a "teacher" to guide a "student" 3D model, such as a Neural Radiance Field (NeRF) or a mesh.

However, standard SDS suffers from a fundamental flaw: it treats all diffusion time-steps equally throughout optimization. In diffusion models, large time-steps correspond to coarse global structures, while small time-steps capture fine-grained textures. By sampling these indiscriminately, the student model often attempts to learn minute details before establishing a stable global shape. This leads to common failure modes: collapsed geometry, the "Janus Face" problem (duplicate features on the back of objects), and over-saturated, "deep-fried" textures.

Methodology: The DTC123 Pipeline

DTC123 introduces a "Diffusion Time-step Curriculum" to transform a single 2D image into a high-quality 3D asset. The pipeline is built on three pillars of a coarse-to-fine curriculum:

  1. Annealed Time-step Schedule: Instead of random sampling, the algorithm begins with large time-steps ($t$) to establish global geometry. As training progresses, it gradually "anneals" (reduces) the time-steps to focus on fine-grained details.
  2. Progressive Student Representation: The 3D model evolves from a low-resolution NeRF using masked hash-encoding to prevent high-frequency noise from ruining the initial shape. In the second stage, it transitions to a high-resolution DMTet (Deep Marching Tetrahedra) for fine-tuning meshes and textures.
  3. Coarse-to-Fine Teacher Collaboration: The system utilizes two distinct teachers. Zero-1-to-3, a viewpoint-conditioned model, provides strong geometric priors during the initial coarse stage. Stable Diffusion is then used in the fine stage for high-fidelity textures, enhanced by LLM-augmented prompts (e.g., GPT-generated descriptions of unseen views) to ensure multi-view consistency.

The pipeline also incorporates "Camera Pose De-biasing" to mitigate the Janus Face problem by clipping gradients at specific azimuth angles where the model is prone to duplicating front-view features onto the back.

Experimental Results

DTC123 was evaluated on standard benchmarks (NeRF4, RealFusion15, GSO) and a new Level50 benchmark containing 50 challenging internet images. The method consistently outperformed baselines like Magic123 and RealFusion.

Quantitative Performance

On the NeRF4 dataset, DTC123 achieved a CLIP-similarity of 0.84 (compared to 0.80 for Magic123) and significantly improved PSNR and LPIPS scores, indicating superior reconstruction fidelity and visual quality.

Qualitative Analysis and Consistency

Qualitatively, DTC123 produces much cleaner geometry and textures. While previous methods often generated extra features (like eyes) on the back of objects, DTC123 maintains consistent surfaces.

Front view of a generated 3D character.

The entire pipeline is efficient, taking approximately 20-25 minutes on a single NVIDIA A100 GPU. Furthermore, by varying the text prompt while keeping the reference image, DTC123 can generate diverse variations of an object while maintaining the original pose.

Conclusion and Future Work

DTC123 demonstrates that the timing of knowledge transfer in SDS is as critical as the prior itself. By scheduling the teacher's supervision from coarse to fine, the method effectively disentangles geometry and texture modeling. While the Janus Face problem is significantly reduced, future work will explore more advanced priors, such as video diffusion models, to further improve multi-view consistency and photo-realism.

Summary updated

Abstract

Score distillation sampling (SDS) has been widely adopted to overcome the absence of unseen views in reconstructing 3D objects from a single image. It leverages pretrained 2D diffusion models as teacher to guide the reconstruction of student 3D models. Despite their remarkable success, SDS-based methods often encounter geometric artifacts and texture saturation. We find out the crux is the overlooked indiscriminate treatment of diffusion time-steps during optimization: it unreasonably treats the student-teacher knowledge distillation to be equal at all time-steps and thus entangles coarse-grained and fine-grained modeling. Therefore, we propose the Diffusion Time-step Curriculum one-image-to-3D pipeline (DTC123), which involves both the teacher and student models collaborating with the time-step curriculum in a coarse-to-fine manner. Extensive experiments on NeRF4, RealFusion15, GSO and Level50 benchmark demonstrate that DTC123 can produce multiview consistent, high-quality, and diverse 3D assets. Codes and more generation demos will be released in https://github.com/yxymessi/DTC123.
BibTeX
@inproceedings{yi2024diffusion,
  author = {Xuanyu Yi and Zike Wu and Qingshan Xu and Pan Zhou and Joo-Hwee Lim and Hanwang Zhang},
  title = {{Diffusion Time-step Curriculum for One Image to 3D Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=h6Es5yeri5},
}
Figure