Repaint123: Fast and High-quality One Image to 3D Generation with Progressive Controllable 2D Repainting

Junwu Zhang, Zhenyu Tang, Yatian Pang, Xinhua Cheng, Peng Jin, Yida Wei, Munan Ning, Li YuanarXiv 2023 · DOI · arXiv

Introduction

Generating high-fidelity 3D models from a single 2D image remains one of the most challenging tasks in computer vision. This "ill-posed" problem requires a system to intelligently hallucinate the unseen sides and textures of an object while maintaining strict consistency with the provided reference view. While recent advancements in 2D diffusion models have provided a powerful prior for this task, existing methods often struggle with a trade-off between generation speed and visual quality.

Repaint123 introduces a novel framework designed to bridge this gap. By moving away from the traditional optimization-heavy paradigms, it achieves high-quality 3D generation in approximately 2 minutes, significantly outperforming previous state-of-the-art methods in both speed and texture detail.

The Limitations of Score Distillation Sampling (SDS)

Most contemporary 3D generation pipelines rely on Score Distillation Sampling (SDS). In an SDS-based workflow, a 2D diffusion model (such as Stable Diffusion) acts as a teacher, providing gradients that guide a 3D representation (like a Neural Radiance Field or NeRF) to look realistic from all possible camera angles.

Despite its popularity, SDS suffers from three critical drawbacks:

  1. Texture Quality: The distilled textures often appear over-saturated, "waxy," or excessively blurry, failing to capture the fine-grained details present in the original image.
  2. Multi-view Inconsistency: SDS frequently falls victim to the "Janus problem," where the model generates conflicting features (e.g., multiple faces or heads) because the 2D prior lacks a global 3D understanding.
  3. Computational Cost: The iterative optimization process is slow, typically requiring 20 minutes to several hours to produce a single model.

Repaint123 addresses these issues by decoupling the 3D generation process into a fast geometry initialization stage followed by a high-quality 2D-driven texture refinement stage.

The Repaint123 Framework

The Repaint123 pipeline operates in two distinct stages: a coarse geometry stage and a progressive refinement stage.

1. Coarse Stage: Rapid Geometry Initialization

To establish a baseline structure, the framework utilizes 3D Gaussian Splatting (3DGS). 3DGS is a point-based rendering technique known for its extreme speed. In roughly one minute, the system generates a coarse 3D representation that captures the basic shape and volume of the target object. This coarse model serves as the geometric foundation for the subsequent refinement.

2. Refining Stage: Progressive Controllable 2D Repainting

This stage represents the core innovation of the paper. Instead of continuing with noisy SDS gradients, the authors convert the 3DGS into a mesh and apply a series of 2D image-to-image enhancements to "repaint" the object's surface with high-resolution details.

  • DDIM Inversion: The system renders a view from the coarse model and "inverts" it back into the diffusion model's latent noise space. This process ensures that the refined image retains the basic color and structural layout of the coarse render.
  • Controllable Denoising: To maintain fidelity to the source image, the framework employs ControlNet (to enforce geometric constraints via depth maps) and IP-Adapter (to use the original image as a visual prompt).
  • Mutual Self-Attention: To ensure that different views of the object share consistent textures, the model utilizes mutual self-attention. This allows the generation of new views to "borrow" features and details directly from the reference image.
  • Visibility-aware Adaptive Repainting: This geometric strategy adjusts the repainting strength based on the camera's angle relative to the surface. If a surface patch is viewed at a sharp, oblique angle, it is likely low-resolution. When the camera moves to a direct, front-on view of that patch, the system increases the repainting strength to effectively "up-sample" the texture detail.

Finally, the framework uses a standard Mean Square Error (MSE) loss to bake these high-quality 2D repainted images back onto the 3D mesh. Because MSE is a deterministic reconstruction loss rather than a stochastic generative one, the refinement happens in seconds and produces much sharper results than SDS.

Experimental Results

Repaint123 was evaluated against several benchmarks, including RealFusion15, Test-alpha, and Google Scanned Objects (GSO). The results demonstrate significant improvements across multiple metrics:

  • Efficiency: The total generation time is approximately 2 minutes, making it 10x to 30x faster than NeRF-based competitors like Magic123.
  • Semantic Fidelity: The framework achieved the highest CLIP similarity scores, indicating that the generated 3D models are more semantically aligned with the input images than those produced by previous methods.
  • Visual Clarity: Qualitative comparisons show that Repaint123 successfully recovers intricate details—such as the texture of feathers or the seeds on a strawberry—that are typically lost or blurred in DreamGaussian or Zero123-XL outputs.

Conclusion and Future Work

Repaint123 demonstrates that high-quality 3D generation does not have to be slow. By prioritizing 2D consistency through progressive repainting and using simple MSE loss for final refinement, the framework bypasses the inherent artifacts of Score Distillation Sampling.

While the texture generation is state-of-the-art, the authors note that the final output is still dependent on the quality of the initial coarse geometry. Future research will likely explore ways to integrate the repainting process more closely with geometry extraction to correct structural errors (such as "holes" in the mesh) during the refinement phase.

Summary updated

Abstract

Recent one image to 3D generation methods commonly adopt Score Distillation Sampling (SDS). Despite the impressive results, there are multiple deficiencies including multi-view inconsistency, over-saturated and over-smoothed textures, as well as the slow generation speed. To address these deficiencies, we present Repaint123 to alleviate multi-view bias as well as texture degradation and speed up the generation process. The core idea is to combine the powerful image generation capability of the 2D diffusion model and the texture alignment ability of the repainting strategy for generating high-quality multi-view images with consistency. We further propose visibility-aware adaptive repainting strength for overlap regions to enhance the generated image quality in the repainting process. The generated high-quality and multi-view consistent images enable the use of simple Mean Square Error (MSE) loss for fast 3D content generation. We conduct extensive experiments and show that our method has a superior ability to generate high-quality 3D content with multi-view consistency and fine textures in 2 minutes from scratch. Our project page is available at https://pku-yuangroup.github.io/repaint123/.
BibTeX
@misc{zhang2023repaint,
  author = {Junwu Zhang and Zhenyu Tang and Yatian Pang and Xinhua Cheng and Peng Jin and Yida Wei and Munan Ning and Li Yuan},
  title = {{Repaint123: Fast and High-quality One Image to 3D Generation with Progressive Controllable 2D Repainting}},
  year = {2023},
  eprint = {2312.13271},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.13271},
}
Figure