DreamLCM: Towards High-Quality Text-to-3D Generation via Latent Consistency Model

Yiming Zhong, Xiaolin Zhang, Yao Zhao, Yunchao Wei

TL;DR

DreamLCM significantly advances text-to-3D generation by replacing standard diffusion guidance with a Latent Consistency Model (LCM). By introducing a guidance calibration strategy and a dual-phase timestep schedule, it produces highly detailed, photo-realistic 3D objects in under an hour—offering a superior balance of quality and efficiency compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

The field of text-to-3D generation has been revolutionized by Score Distillation Sampling (SDS), a technique that uses pre-trained 2D diffusion models to "guide" the creation of 3D assets. However, SDS-based methods frequently suffer from the "over-smooth issue," where generated objects appear blurry, lack fine texture, and look "feature-averaged."

The authors identify two primary culprits for this:

  1. Low-Quality Guidance: Standard diffusion models (like Stable Diffusion) produce noisy, blurred gradients when performing the single-step inference required for SDS.
  2. Inconsistency: The inherent randomness of input noise and timesteps during the optimization process causes the 3D model to essentially "average out" different versions of the object, washing away intricate details.

Methods

DreamLCM tackles these issues by rethinking the guidance mechanism and the optimization schedule:

  • LCM Integration: Instead of standard diffusion, the authors use a Latent Consistency Model (LCM). LCMs are designed to map any point on a probability flow trajectory to its origin in a single step. This property allows DreamLCM to generate much sharper and more consistent guidance gradients compared to traditional models.
  • Guidance Calibration: Even with LCM, single-step predictions can be slightly off. The authors propose a two-stage calibration strategy using an Euler Solver. In the refinement phase, they perturb the rendered image, predict guidance, and then use the numerical ODE solver to "calibrate" the distribution, pushing the optimization direction closer to the high-density data region (i.e., making it more realistic).
  • Dual Timestep Strategy: The optimization is split into two distinct phases:
  • Initial Phase (Geometry): Uses large timesteps to encourage large-scale deformations and establish the global structure of the 3D object.
  • Refinement Phase (Appearance): Uses smaller, monotonically decreasing timesteps to focus on fine-grained textures and local details.
  • 3D Representation: The method utilizes 3D Gaussian Splatting (3DGS), initialized by point-cloud models like Point-E or Shap-E, which allows for faster rendering and optimization than traditional Neural Radiance Fields (NeRF).
Illustration of the DreamLCM framework, showing the dual-phase training and guidance calibration.

Datasets

As a distillation-based method, DreamLCM does not require a new 3D training dataset. Instead, it leverages:

  • LAION-5B: The massive 2D image-text dataset used to pre-train the underlying Latent Consistency Model.
  • Point-E / Shap-E: Pre-trained 3D point cloud models used for initial geometry setup.
  • Qualitative Benchmarks: Evaluation is performed across a wide variety of complex text prompts to test creativity, consistency, and detail (e.g., "A portrait of a unicorn," "A Spanish galleon").

Key Insights and Ideas

  • Consistency over Randomness: The core "aha moment" is that the over-smoothing in SDS isn't just about noise—it's about inconsistent noise. By using LCM's consistency property, the model receives a stable "target" to optimize toward, preventing the averaging effect.
  • Mathematical Calibration: Using an Euler Solver to refine guidance is a clever way to bridge the gap between fast single-step inference and high-quality multi-step denoising.
  • Decoupling Geometry and Texture: By tying the timestep schedule to specific optimization goals (large timesteps for shape, small for skin/fur), the authors provide a more stable path to convergence.

Key Results

DreamLCM achieves state-of-the-art results in both visual fidelity and training speed:

  • High Fidelity: As shown in Figure 3 and 4, the model generates intricate details like silky fur on a unicorn or fine wood grain on a galleon that previous methods struggle to capture.
  • Efficiency: DreamLCM takes approximately 50 minutes to generate a high-quality sample on a single NVIDIA RTX 3090 GPU. This is significantly faster than ProlificDreamer, which can take up to 10 hours, while producing comparable or superior detail.
  • Ablation Success: Experiments confirm that each component—LCM guidance, the dual timestep strategy, and guidance calibration—is essential for reaching the final level of sharpness.
Comparison of DreamLCM against other state-of-the-art methods, highlighting the superior detail and consistency.

Limitations and Future Work

  • Initialization Sensitivity: Like many Gaussian Splatting methods, DreamLCM relies on Point-E or Shap-E for a "warm start." If the initial point cloud is poor (e.g., for very unusual prompts), the final 3D model may inherit those geometric flaws.
  • Multi-face (Janus) Problem: While the increased consistency helps, the method does not explicitly solve the "Janus problem" (where an object might have multiple faces) common in text-to-3D, though it shows empirical improvements.
  • Future Directions: The authors suggest that the principles of DreamLCM could be extended to 4D (video/dynamic 3D) generation or integrated with even more advanced numerical solvers for further speedups.
Examples of various 3D models generated by DreamLCM, demonstrating its versatility across different categories.

Summary updated

Abstract

Recently, the text-to-3D task has developed rapidly due to the appearance of the SDS method. However, the SDS method always generates 3D objects with poor quality due to the over-smooth issue. This issue is attributed to two factors: 1) the DDPM single-step inference produces poor guidance gradients; 2) the randomness from the input noises and timesteps averages the details of the 3D contents. In this paper, to address the issue, we propose DreamLCM which incorporates the Latent Consistency Model (LCM). DreamLCM leverages the powerful image generation capabilities inherent in LCM, enabling generating consistent and high-quality guidance, i.e., predicted noises or images. Powered by the improved guidance, the proposed method can provide accurate and detailed gradients to optimize the target 3D models. In addition, we propose two strategies to enhance the generation quality further. Firstly, we propose a guidance calibration strategy, utilizing Euler Solver to calibrate the guidance distribution to accelerate 3D models to converge. Secondly, we propose a dual timestep strategy, increasing the consistency of guidance and optimizing 3D models from geometry to appearance in DreamLCM. Experiments show that DreamLCM achieves state-of-the-art results in both generation quality and training efficiency. The code is available at https://github.com/1YimingZhong/DreamLCM.
BibTeX
@inproceedings{zhong2024dreamlcm,
  author = {Yiming Zhong and Xiaolin Zhang and Yao Zhao and Yunchao Wei},
  title = {{DreamLCM: Towards High-Quality Text-to-3D Generation via Latent Consistency Model}},
  booktitle = {ACM Multimedia},
  year = {2024},
}
Figure