DreamTexture: Shape from Virtual Texture with Analysis by Augmentation

Ananta R. Bhattarai, Xingzhe He, Alla Sheffer, Helge RhodinarXiv 2025 · arXiv

TL;DR

DreamTexture introduces "Analysis by Augmentation," a novel paradigm that reconstructs 3D shape from a single image by superimposing a virtual texture and aligning it using 2D diffusion priors. By replacing expensive multi-view volumetric rendering with conformal map optimization, it achieves highly efficient, unsupervised single-view 3D reconstruction with a fraction of the parameters of previous methods.

What Problem Does This Paper Address?

Unsupervised 3D reconstruction from 2D images has advanced significantly thanks to Score Distillation Sampling (SDS), popularized by frameworks like DreamFusion. However, existing state-of-the-art methods rely on Analysis by Synthesis: they render multiple virtual views of a 3D representation (like a NeRF or Gaussian Splat) and optimize it so that every view looks plausible to a pre-trained diffusion model.

This multi-view rendering process is computationally expensive, memory-intensive, and often under-constrained when starting from a single image. DreamTexture addresses this bottleneck by asking a fundamental question: Can we leverage the monocular geometric understanding of 2D diffusion models directly from a single viewpoint, without the need for costly multi-view synthesis?

Figure 1. Teaser. While existing analysis-by-synthesis approaches (left) rely on volumetric rendering of multiple real or virtual views, DreamTexture (right) takes one image as input, augments a virtual texture, and optimizes the output depth map by aligning the real and virtual shape cues with a pre-trained image prior via Score Distillation Sampling (SDS). This analysis by augmentation (AbA) approach reconstructs depth without any 3D supervision and unlike classical shape from texture, it applies to both textured and textureless objects.

Methods

Instead of generating new viewpoints, DreamTexture introduces Analysis by Augmentation (AbA). The core idea is to superimpose a regular virtual texture (like a checkerboard pattern) onto the input image and optimize its alignment based on the image's inherent shape cues.

The pipeline operates in two distinct stages:

  1. Stage I: Texture Optimization: A flat, regular checkerboard texture is alpha-blended onto the input image. Using SDS loss from a pre-trained diffusion model (Stable Diffusion v1.5), the system warps the 2D texture coordinates until the pattern conforms naturally to the underlying object's perspective, shading, and silhouette cues.
  2. Stage II: Depth Optimization: To translate these 2D texture distortions into 3D shape, the authors creatively repurpose a classic computer graphics algorithm: Least Squares Conformal Mapping (LSCM). While LSCM is traditionally used to flatten a 3D mesh into a 2D texture map with minimal angular distortion, DreamTexture runs it in reverse. By freezing the optimized 2D texture coordinates from Stage I, they optimize the depth ($z$-coordinates) of a 3D mesh grid to minimize the texture's angular distortion in 3D space.
Figure 3. DreamTexture overview. In Stage I, we use the SDS loss to faithfully augment the texture on the input image, producing texture coordinates that align the virtual texture with depth cues in the input image. In Stage II, we apply shape from virtual texture by viewing the depth map as the z-coordinates of a 3D mesh and optimize it using the LSCM energy, minimizing angular texture distortion.

Datasets

The authors evaluated their approach using both synthetic benchmarks and real-world images:

  • PrimitiveShapesX: A newly constructed synthetic dataset generated in Blender. It features four basic geometric primitives (cube, pyramid, cylinder, and sphere) rendered under three distinct surface conditions: shading only (textureless), regular repetitive texture, and natural wood texture. This allows for isolated testing of how different monocular cues impact reconstruction.
  • In-the-Wild Images: Real-world, single-view images collected from the internet to test the model's generalization capability.

Key Insights and Ideas

  • Diffusion Models as Monocular Geometric Priors: The paper demonstrates that 2D generative models possess a strong implicit understanding of monocular shape cues (like shading and perspective). This knowledge can be extracted directly through texture alignment rather than novel view synthesis.
  • Inverting LSCM for Depth Inference: Flipping LSCM on its head is a powerful conceptual shift. Instead of finding texture coordinates for a known shape, the method finds the shape that explains the distorted texture coordinates.
  • Massive Parameter Efficiency: By bypassing volumetric representations (NeRFs/Gaussian Splats) and directly optimizing a single-view depth mesh, the method slashes the number of trainable parameters from millions down to just a few thousand.

Key Results

  • Superior Accuracy on Featureless Objects: DreamTexture significantly outperforms multi-view baselines like RealFusion on simple or untextured shapes where multi-view constraints typically fail due to a lack of distinct visual features. For instance, on a synthetic cube, DreamTexture achieved a Depth Mean Squared Error (MSE) of 0.0071 compared to RealFusion's 0.0706.
  • Drastic Parameter Reduction: As shown in the paper's efficiency analysis, DreamTexture requires only 15,018 trainable parameters in total, compared to over 12.6 million for DreamFusion and nearly 40 million for RealFusion.
Figure 4. Text-to-3D comparison. Compared to RealFusion (RF) and DreamFusion (DF), our method achieves more accurate reconstruc- tions, particularly for objects that lack prominent visual features. We use the same text prompt as DreamFusion to generate input images with Stable Diffusion. To render novel views in our approach, we resample the input image using the inverse texture coordinates and render using PyTorch3D with orthographic projection (elevation = 0◦) and azimuth angles of 0◦(view 1), −10◦(view 2), and +10◦(view 3).

Limitations and Future Work

While promising, this first realization of Analysis by Augmentation has notable limitations:

  • 2.5D Reconstruction: Because it operates from a single viewpoint, it only reconstructs the visible surface (a depth map) rather than a full 360-degree 3D asset.
  • Complex Geometries: The method struggles with highly complex topologies, such as cars or fine-grained details, partly due to the low resolution of the optimized texture coordinates.
  • Concave/Convex Ambiguity: The pipeline can fail to distinguish between concave and convex structures, occasionally reconstructing a cup as a solid convex object.

The authors suggest that future iterations could overcome these challenges by partitioning complex images into parts or combining this monocular approach with sparse multi-view constraints.

Summary updated

Abstract

DreamFusion established a new paradigm for unsupervised 3D reconstruction from virtual views by combining advances in generative models and differentiable rendering. However, the underlying multi-view rendering, along with supervision from large-scale generative models, is computationally expensive and under-constrained. We propose DreamTexture, a novel Shape-from-Virtual-Texture approach that leverages monocular depth cues to reconstruct 3D objects. Our method textures an input image by aligning a virtual texture with the real depth cues in the input, exploiting the inherent understanding of monocular geometry encoded in modern diffusion models. We then reconstruct depth from the virtual texture deformation with a new conformal map optimization, which alleviates memory-intensive volumetric representations. Our experiments reveal that generative models possess an understanding of monocular shape cues, which can be extracted by augmenting and aligning texture cues -- a novel monocular reconstruction paradigm that we call Analysis by Augmentation.
BibTeX
@misc{bhattarai2025dreamtexture,
  author = {Ananta R. Bhattarai and Xingzhe He and Alla Sheffer and Helge Rhodin},
  title = {{DreamTexture: Shape from Virtual Texture with Analysis by Augmentation}},
  year = {2025},
  eprint = {2503.16412},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.16412},
}
Figure