Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation

Wangguandong Zheng, Haifeng Xia, Rui Chen, Libo Sun, Ming Shao, Siyu Xia, Zhengming Ding

Introduction

In the early stages of creative workflows—ranging from industrial design to game development—artists often rely on rough sketches to conceptualize ideas. While recent advancements in image-to-3D generation have been significant, they typically require high-quality, "natural" images as input. Translating a sparse, abstract 2D sketch into a high-fidelity 3D asset remains a significant challenge due to information scarcity and the difficulty of maintaining structural consistency across multiple views.

Sketch3D addresses these hurdles by introducing a novel framework that transforms simple 2D sketches and text prompts into high-fidelity 3D assets in approximately three minutes. By leveraging 3D Gaussian Splatting (3DGS) and a multi-stage optimization process, the system ensures that the resulting model remains geometrically faithful to the original sketch while adopting the stylistic and color attributes defined by the user's text prompt.

The Sketch3D Framework

The Sketch3D pipeline follows a structured "Generate-Initialize-Optimize" workflow designed to bridge the gap between abstract sketches and detailed 3D geometry.

1. Shape-Preserving Reference Generation

The process begins by generating a high-quality 2D reference image that serves as a visual anchor. Using ControlNet conditioned on the input sketch and Stable Diffusion, the system produces an image that adheres to the sketch's structural lines while incorporating the textures and colors requested in the text prompt. This step effectively "hallucinates" the missing details required for 3D reconstruction.

2. Gaussian Initialization

To accelerate the 3D generation process, the framework initializes a coarse 3D representation using 3D Gaussian Splatting (3DGS). A pre-trained 3D diffusion model, such as Shap-E, converts the 2D reference image into a rough 3D point cloud, which is then transformed into a set of 3D Gaussians. This provides a better starting point for optimization than random initialization.

3. Style-Consistent Optimization

The core innovation of Sketch3D lies in its optimization stage. The system utilizes an IP-Adapter to generate multi-view guidance images that are stylistically consistent with the initial reference image. Three specific optimization strategies are then applied:

  • Structural Optimization: A distribution transfer mechanism aligns the 3D structure with the statistical distribution of the guidance images, avoiding the "blurry" artifacts often associated with standard Score Distillation Sampling (SDS).
  • Color Optimization: A Mean Squared Error (MSE) loss is employed to refine textures and ensure color consistency against the multi-view guidance.
  • Sketch Similarity Optimization: To prevent the model from drifting away from the original drawing, the authors implement a CLIP-based geometric loss. By comparing mid-level activations (specifically Layer 4 of a ResNet101 CLIP model) of the sketch and the rendered 3D views, the system captures geometric similarity rather than just high-level semantic meaning.

The ShapeNet-Sketch3D Benchmark

To rigorously evaluate the framework, the authors introduced ShapeNet-Sketch3D, a new benchmark derived from the classic ShapeNet dataset. This dataset includes 10 categories with 1,100 objects each, featuring multi-view renderings, synthetic sketches extracted via Canny edge detection, and detailed textual descriptions generated by GPT-4-vision. This provides a standardized environment for testing the alignment between sketch, text, and 3D ground truth.

Key Insights and Results

Sketch3D demonstrates significant improvements in both efficiency and quality over existing methods:

  • Efficiency: By utilizing 3D Gaussian Splatting instead of Neural Radiance Fields (NeRFs), the model achieves rapid rendering and optimization, completing a generation in roughly 3 minutes on a single RTX 4090.
  • Quantitative Performance: Sketch3D outperformed five major baselines, including DreamGaussian and Shap-E, achieving a CLIP-Similarity of 0.779 and an SSIM of 0.818. These metrics indicate superior alignment with both the textual prompt and the visual reference.
  • Qualitative Superiority: Visual results show that the framework effectively avoids the "Janus problem" (multi-face artifacts) common in 3D generation. It maintains high fidelity whether the input is a clean synthetic sketch or a messy, hand-drawn doodle.
  • User Validation: In a study involving 9 participants rating 50 generated assets, Sketch3D received the highest scores for both "Fidelity" (4.12/5) and "Consistency" (3.94/5).

Limitations and Future Directions

Despite its performance, Sketch3D is heavily dependent on the quality of the initial 2D reference image generated by ControlNet. If the initial image contains structural flaws, those errors are likely to propagate into the 3D model. Additionally, while the model excels at general shapes, it can struggle with extremely intricate sketches where every minute line is critical to the design. Future work will likely focus on reducing the dependency on the initial 2D generation step and achieving even finer control over micro-details.

Summary updated

BibTeX
@inproceedings{zheng2024sketch,
  author = {Wangguandong Zheng and Haifeng Xia and Rui Chen and Libo Sun and Ming Shao and Siyu Xia and Zhengming Ding},
  title = {{Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3680641},
}
Figure