Chasing Consistency in Text-to-3D Generation from a Single Image

Yichen Ouyang, Wenhao Chai, Jiayi Ye, Dapeng Tao, Yibing Zhan, Gaoang WangarXiv 2023 · OpenReview

Abstract

Text-to-3D generation from a single-view image is a popular but challenging task in 3D vision. Although numerous methods have been proposed, existing works still suffer from inconsistency issues, including 1) semantic inconsistency, 2) geometric inconsistency, and 3) saturation inconsistency, resulting in distorted, overfitted, and over-saturated generations. In light of these issues, we present Consist3D, a three-stage framework chasing semantic-, geometric-, and saturation-consistent text-to-3D generation. The first two stages aim to learn parameterized consistency tokens, while the last stage focuses on optimization. Specifically, the semantic encoding stage learns a token independent of views, promoting robustness. The geometric encoding stage learns another token with comprehensive geometry constraints, reducing overfitting. Finally, the optimization stage benefits from these tokens, allowing a low classifier-free guidance scale to prevent oversaturation. Experimental results demonstrate that Consist3D produces more consistent, faithful, and photo-realistic 3D assets compared to previous state-of-the-art methods.

Addressing the Inconsistency Gap

Generating a 3D object from a single 2D image is a "holy grail" task in computer vision, but current methods frequently fail in three specific ways that degrade the quality of the output:

  1. Semantic Inconsistency: The model loses track of the object's identity when viewed from novel angles (e.g., a hat might morph into a box).
  2. Geometric Inconsistency: Often called the "Janus problem," models tend to overfit to the reference view, resulting in artifacts like a second face appearing on the back of an animal.
  3. Saturation Inconsistency: To maintain stability, existing methods use high Classifier-Free Guidance (CFG) scales, resulting in "deep-fried" or over-saturated colors that do not match the original image.
The three primary inconsistency issues in single-image-to-3D generation: semantic, geometric, and saturation.

The Consist3D Framework

Consist3D addresses these issues through a structured, three-stage pipeline that moves away from brute-force optimization toward a more stable, token-based approach:

  • Stage I: Semantic Encoding: The model learns a parameterized semantic token using a DreamBooth-like fine-tuning process. This token captures the core identity of the object independent of the camera view, providing a robust anchor for the generation.
  • Stage II: Geometric Encoding: A second geometric token is trained using a "Warper" module. This module utilizes depth maps (estimated via Point-E) to ensure pixels move correctly between neighboring views. By applying Warp and Reconstruction losses, the model learns a consistent 3D shape that maintains integrity during camera rotation.
  • Stage III: Low-scale Score Distillation Sampling (SDS): Because the semantic and geometric tokens provide a strong foundation, the final optimization can use a much lower CFG scale (under 25). This prevents the over-saturation typical of previous methods while maintaining 3D stability.

Crucially, because the system is grounded in text tokens, it allows for seamless editing. Users can change backgrounds or modify the object itself through simple text prompts while preserving the underlying 3D structure.

Experimental Results

Consist3D was evaluated on a diverse range of subjects, including in-the-wild images from ImageNet, cartoon characters, and the DreamBooth3D dataset. Quantitative results show that Consist3D significantly outperforms baselines like DreamFusion and 3DFuse, achieving an average CLIP Score of 78.06 compared to 70.71 and 70.46, respectively.

Multi-view generation demonstrating geometric consistency and lack of Janus artifacts.

In a user study with 137 participants, Consist3D was overwhelmingly preferred, scoring 89.73% in saturation consistency and 85.35% in semantic consistency—nearly 20 points higher than its closest competitors. Visual results demonstrate that Consist3D produces objects with realistic backsides and colors that remain faithful to the original reference image.

Conclusion and Future Work

Consist3D demonstrates that high CFG scales are often a "crutch" used to compensate for poor consistency. By fixing consistency issues in the encoding stages, models can return to natural color levels. While the method currently relies on initial 3D point cloud estimations (Point-E) which can introduce distortions if the initial estimate is poor, it represents a significant step forward. Future work will explore modeling objects and backgrounds as separate entities to allow for even finer control and higher resolution in the final 3D assets.

Summary updated

Abstract

Text-to-3D generation from a single-view image is a popular but challenging task in 3D vision. Although numerous methods have been proposed, existing works still suffer from the inconsistency issues, including 1) semantic inconsistency, 2) geometric inconsistency, and 3) saturation inconsistency, resulting in distorted, overfitted, and over-saturated generations. In light of the above issues, we present Consist3D, a three-stage framework Chasing for semantic-, geometric-, and saturation-Consistent Text-to-3D generation from a single image, in which the first two stages aim to learn parameterized consistency tokens, and the last stage is for optimization. Specifically, the semantic encoding stage learns a token independent of views and estimations, promoting semantic consistency and robustness. Meanwhile, the geometric encoding stage learns another token with comprehensive geometry and reconstruction constraints under novel-view estimations, reducing overfitting and encouraging geometric consistency. Finally, the optimization stage benefits from the semantic and geometric tokens, allowing a low classifier-free guidance scale and therefore preventing oversaturation. Experimental results demonstrate that Consist3D produces more consistent, faithful, and photo-realistic 3D assets compared to previous state-of-the-art methods. Furthermore, Consist3D also allows background and object editing through text prompts.
BibTeX
@misc{ouyang2023chasing,
  author = {Yichen Ouyang and Wenhao Chai and Jiayi Ye and Dapeng Tao and Yibing Zhan and Gaoang Wang},
  title = {{Chasing Consistency in Text-to-3D Generation from a Single Image}},
  year = {2023},
  eprint = {2309.03599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.03599},
}
Figure