Abstract
Reconstructing 3D objects from single images typically requires clean backgrounds or manual silhouette annotations to bridge the gap between real-world photos and simplified computer renderings. This paper presents a novel approach using an Adversarial Style-Transfer Autoencoder (ASTA) to perform unsupervised domain adaptation. By co-evolving a 3D reconstruction network with a style-transfer pipeline, the system learns to translate "messy" real-world images into a simplified style compatible with differentiable renderers. This allows for accurate 3D geometry extraction from complex, textured scenes without the need for manual silhouette supervision.
Introduction
Single-view 3D reconstruction—the ability to generate a 3D model from a single 2D photograph—is a fundamental challenge in computer vision. While recent advances in differentiable rendering have made significant strides, they often rely on a "clean" environment: smooth surfaces, uniform lighting, and white backgrounds. Real-world imagery is rarely so cooperative, featuring complex textures and busy backgrounds that create a significant "domain gap" for standard reconstruction models. This paper introduces a framework that bridges this gap by making the real world look like a simple rendering, rather than trying to make the renderer simulate the entire real world.
Methodology
The proposed framework consists of two interlocking pipelines that are trained together in an interleaved, "co-evolutionary" fashion.
The Inverse Rendering Pipeline
At the heart of the reconstruction is a 3D reconstruction network ($R$) that predicts a mesh from an input image. This mesh is then processed by a Smooth Differentiable Renderer ($SDR$) to generate a 2D "synthetic" image. The goal is to minimize the difference between this synthetic output and a stylized version of the original input.
The ASTA Pipeline
The Adversarial Style-Transfer Autoencoder (ASTA) acts as the bridge between the real and synthetic domains. It performs image-to-image translation through three main components:
- Encoder ($F_{i2r}$): Translates the complex input image into the simplified "rendered" domain.
- Decoder ($F_{r2i}$): Translates the image back to the original domain to ensure content preservation via cycle-consistency loss.
- Discriminator ($D$): An adversarial component that forces the encoder's output to be stylistically indistinguishable from the outputs of the differentiable renderer.
To ensure training stability and prevent the style transfer from collapsing into a trivial solution, the authors employ a Jaccard similarity loss. This loss keeps the translated image aligned with the current 3D prediction, providing a consistent signal for the reconstruction network.
Experimental Setup and Datasets
The model was evaluated using three distinct datasets to test its robustness across varying levels of complexity:
- ShapeNet: The standard benchmark for 3D object reconstruction.
- ShapeNet with Minecraft Backgrounds: A custom dataset where ShapeNet objects are rendered against 135 different Minecraft scenes to simulate complex, textured real-world backgrounds.
- ShapeNet2Lego: A real-world benchmark consisting of physical LEGO models photographed against LEGO-brick backgrounds, paired with ground-truth CAD models for evaluation.
Results and Evaluation
The proposed method demonstrates superior performance in handling textured backgrounds compared to existing weakly-supervised methods. On the ShapeNet-Minecraft dataset, the model achieved a mean Intersection-over-Union (IoU) of 0.3816, significantly outperforming baselines that had access to silhouette supervision (0.2563).
In simpler scenarios with uniform backgrounds, the method reached an accuracy of 57.4%, rivaling established methods that require explicit silhouette labels. Furthermore, the model showed strong generalization capabilities; when trained on synthetic data, it successfully reconstructed 3D shapes from real-world ShapeNet2Lego photos, achieving an average IoU of 0.3316 despite the domain shift from Minecraft textures to physical LEGO bricks.
Discussion: Co-evolutionary Training
A key insight of this work is the "co-evolution" of the two pipelines. Initially, the 3D reconstruction network produces primitive shapes (like spheres). As the ASTA pipeline improves its ability to isolate the object from the background, the reconstruction network receives a cleaner signal to refine the 3D geometry. Conversely, as the 3D reconstruction becomes more accurate, the ASTA pipeline receives better examples of what a "correct" rendering should look like, creating a virtuous cycle of improvement.
Conclusion and Future Work
This research provides a robust path forward for "in-the-wild" 3D reconstruction by removing the dependency on manual silhouette annotations. While the current model requires known camera extrinsics and focuses on single-object scenes, it demonstrates remarkable resilience to camera noise and background complexity. Future work will likely focus on extending this framework to multi-object scenes with occlusions and exploring category-specific priors to improve initial geometry estimation.
Summary updated