Diffusion Time-step Curriculum for One Image to 3D Generation

Xuanyu Yi, Zike Wu, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, Hanwang Zhang

Introduction

Generating high-fidelity 3D assets from a single 2D image is a cornerstone challenge in computer vision. While Score Distillation Sampling (SDS) has emerged as a leading technique—leveraging pre-trained 2D diffusion models to guide 3D representations—it often suffers from significant artifacts. These include the "Janus-faced" problem (multiple faces on a single object) and oversaturated or blurry textures. These issues stem from a fundamental lack of coordination in the optimization process.

This paper introduces DTC123, a framework that implements a Diffusion Time-step Curriculum. By strategically scheduling the optimization process to prioritize coarse geometry before fine-grained details, DTC123 produces multi-view consistent, high-quality 3D models in approximately 20-25 minutes.

The Problem: Time-step Entanglement

In standard SDS, the optimization process samples diffusion time-steps ($t$) uniformly. In the context of diffusion models, large values of $t$ correspond to high noise levels and represent global, coarse structures, while small values of $t$ represent low noise and fine-grained details.

When these time-steps are sampled randomly throughout training, the model attempts to learn intricate details (like eyelashes or skin pores) simultaneously with the global structure (the overall shape of a head). This "entanglement" leads to geometric instability. If the model attempts to refine details before a coherent global structure is established, the resulting 3D object often lacks spatial consistency, leading to the notorious Janus Face artifact.

The DTC123 Methodology

DTC123 addresses these challenges through a three-pronged curriculum strategy:

1. Annealed Time-step Schedule

The core innovation is an annealing schedule for diffusion time-steps. The authors provide a mathematical foundation (Theorem 1) demonstrating a lower bound for time-steps in the early stages of SDS. If $t$ is too small too early, the teacher model provides noisy, invalid guidance because the student representation hasn't yet formed a basic shape. DTC123 starts by sampling large $t$ values to establish global geometry and gradually narrows the window toward smaller $t$ values to refine textures and surfaces.

2. Progressive 3D Representation

The "student" model evolves alongside the curriculum:

  • Stage 1 (Coarse): The model uses a low-resolution NeRF (Neural Radiance Field) with hash-grid encoding to capture basic volumetric shapes.
  • Stage 2 (Fine): The representation is converted into a high-resolution DMTet (a deformable tetrahedral grid). This allows for explicit mesh extraction and high-fidelity texture refinement that NeRFs often struggle to achieve efficiently.

3. Coarse-to-Fine Teacher Guidance

DTC123 utilizes a "tag-team" of teacher models specialized for different tasks:

  • Zero-1-to-3: A viewpoint-conditioned model that excels at maintaining geometric structure. This serves as the primary guide during the early, coarse stage.
  • Stable Diffusion: Used in the later stages to provide high-fidelity texture details. To ensure consistency across views, the authors use LLM-augmented prompts. GPT-4 is employed to expand a simple prompt (e.g., "a robot") into consistent descriptions for the front, back, and sides, reducing the ambiguity that typically confuses the diffusion process.

To further mitigate the Janus Face problem, the pipeline implements camera pose de-biasing through dynamic gradient clipping and randomized dropout for specific azimuth ranges, preventing the model from erroneously duplicating the front view onto the back of the object.

Experimental Results

The researchers evaluated DTC123 against state-of-the-art baselines, including Magic123 and RealFusion, across several benchmarks: NeRF4, RealFusion15, Google Scanned Objects (GSO), and a new "Level50" benchmark consisting of 50 diverse, difficult real-world images.

Quantitative Performance

On the NeRF4 dataset, DTC123 demonstrated superior performance across key metrics:

  • CLIP-Similarity: 0.84 (compared to Magic123’s 0.80).
  • PSNR: 25.14.
  • LPIPS: 0.02.

These metrics indicate not only higher image quality but also a more faithful reconstruction of the reference view compared to previous methods.

Qualitative and Efficiency Gains

Qualitatively, DTC123 produces significantly sharper textures and more robust geometry. Failure rate analysis on the Level50 benchmark showed that DTC123 is less sensitive to random initializations than its competitors. Furthermore, the entire optimization pipeline is efficient, running in 20-25 minutes on a single NVIDIA A100 GPU.

Conclusion and Future Work

DTC123 demonstrates that a structured curriculum is essential for stable 1-image-to-3D generation. By decoupling geometry and texture optimization through time-step annealing and progressive representations, it overcomes the primary pitfalls of SDS.

While highly effective, the framework's quality remains capped by the underlying 2D diffusion teachers. Future improvements in models like Stable Diffusion 3 or more advanced viewpoint-conditioned models will likely further enhance DTC123's output. Additionally, while 20 minutes is a significant improvement for optimization-based methods, the gap between these high-quality results and near-instant "feed-forward" models remains an active area for research. Handling complex materials, such as those with high transparency or reflectivity, also remains a challenge for future iterations.

Summary updated

BibTeX
@inproceedings{yi2024diffusion,
  author = {Xuanyu Yi and Zike Wu and Qingshan Xu and Pan Zhou and Joo-Hwee Lim and Hanwang Zhang},
  title = {{Diffusion Time-step Curriculum for One Image to 3D Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=h6Es5yeri5},
}
Figure