Style Agnostic 3D Reconstruction via Adversarial Style Transfer

Felix Petersen, Bastian Goldluecke, Oliver Deussen, Hilde Kuehne

Abstract

Reconstructing 3D objects from single images typically requires clean backgrounds or manual silhouette annotations to bridge the gap between real-world photos and simplified computer renderings. This paper presents a novel approach using an Adversarial Style-Transfer Autoencoder (ASTA) to perform unsupervised domain adaptation. By co-evolving a 3D reconstruction network with a style-transfer pipeline, the system learns to translate "messy" real-world images into a simplified style compatible with differentiable renderers. This allows for accurate 3D geometry extraction from complex, textured scenes without the need for manual silhouette supervision.

Introduction

Single-view 3D reconstruction—the ability to generate a 3D model from a single 2D photograph—is a fundamental challenge in computer vision. While recent advances in differentiable rendering have made significant strides, they often rely on a "clean" environment: smooth surfaces, uniform lighting, and white backgrounds. Real-world imagery is rarely so cooperative, featuring complex textures and busy backgrounds that create a significant "domain gap" for standard reconstruction models. This paper introduces a framework that bridges this gap by making the real world look like a simple rendering, rather than trying to make the renderer simulate the entire real world.

Methodology

The proposed framework consists of two interlocking pipelines that are trained together in an interleaved, "co-evolutionary" fashion.

The Inverse Rendering Pipeline

At the heart of the reconstruction is a 3D reconstruction network ($R$) that predicts a mesh from an input image. This mesh is then processed by a Smooth Differentiable Renderer ($SDR$) to generate a 2D "synthetic" image. The goal is to minimize the difference between this synthetic output and a stylized version of the original input.

The ASTA Pipeline

The Adversarial Style-Transfer Autoencoder (ASTA) acts as the bridge between the real and synthetic domains. It performs image-to-image translation through three main components:

  • Encoder ($F_{i2r}$): Translates the complex input image into the simplified "rendered" domain.
  • Decoder ($F_{r2i}$): Translates the image back to the original domain to ensure content preservation via cycle-consistency loss.
  • Discriminator ($D$): An adversarial component that forces the encoder's output to be stylistically indistinguishable from the outputs of the differentiable renderer.
The proposed framework architecture, featuring the Adversarial Style-Transfer Autoencoder (ASTA) and the Inverse Rendering Pipeline.

To ensure training stability and prevent the style transfer from collapsing into a trivial solution, the authors employ a Jaccard similarity loss. This loss keeps the translated image aligned with the current 3D prediction, providing a consistent signal for the reconstruction network.

Experimental Setup and Datasets

The model was evaluated using three distinct datasets to test its robustness across varying levels of complexity:

  1. ShapeNet: The standard benchmark for 3D object reconstruction.
  2. ShapeNet with Minecraft Backgrounds: A custom dataset where ShapeNet objects are rendered against 135 different Minecraft scenes to simulate complex, textured real-world backgrounds.
  3. ShapeNet2Lego: A real-world benchmark consisting of physical LEGO models photographed against LEGO-brick backgrounds, paired with ground-truth CAD models for evaluation.

Results and Evaluation

The proposed method demonstrates superior performance in handling textured backgrounds compared to existing weakly-supervised methods. On the ShapeNet-Minecraft dataset, the model achieved a mean Intersection-over-Union (IoU) of 0.3816, significantly outperforming baselines that had access to silhouette supervision (0.2563).

Qualitative results showing the translation from textured input images to simplified 'rendered' styles and the resulting 3D reconstructions.

In simpler scenarios with uniform backgrounds, the method reached an accuracy of 57.4%, rivaling established methods that require explicit silhouette labels. Furthermore, the model showed strong generalization capabilities; when trained on synthetic data, it successfully reconstructed 3D shapes from real-world ShapeNet2Lego photos, achieving an average IoU of 0.3316 despite the domain shift from Minecraft textures to physical LEGO bricks.

Discussion: Co-evolutionary Training

A key insight of this work is the "co-evolution" of the two pipelines. Initially, the 3D reconstruction network produces primitive shapes (like spheres). As the ASTA pipeline improves its ability to isolate the object from the background, the reconstruction network receives a cleaner signal to refine the 3D geometry. Conversely, as the 3D reconstruction becomes more accurate, the ASTA pipeline receives better examples of what a "correct" rendering should look like, creating a virtuous cycle of improvement.

Conclusion and Future Work

This research provides a robust path forward for "in-the-wild" 3D reconstruction by removing the dependency on manual silhouette annotations. While the current model requires known camera extrinsics and focuses on single-object scenes, it demonstrates remarkable resilience to camera noise and background complexity. Future work will likely focus on extending this framework to multi-object scenes with occlusions and exploring category-specific priors to improve initial geometry estimation.

Summary updated

Abstract

Reconstructing the 3D geometry of an object from an image is a major challenge in computer vision. Recently introduced differentiable renderers can be leveraged to learn the 3D geometry of objects from 2D images, but those approaches require additional supervision to enable the renderer to produce an output that can be compared to the input image. This can be scene information or constraints such as object silhouettes, uniform backgrounds, material, texture, and lighting. In this paper, we propose an approach that enables a differentiable rendering-based learning of 3D objects from images with backgrounds without the need for silhouette supervision. Instead of trying to render an image close to the input, we propose an adversarial style-transfer and domain adaptation pipeline that allows to translate the input image domain to the rendered image domain. This allows us to directly compare between a translated image and the differentiable rendering of a 3D object reconstruction in order to train the 3D object reconstruction network. We show that the approach learns 3D geometry from images with backgrounds and provides a better performance than constrained methods for single-view 3D object reconstruction on this task.
BibTeX
@inproceedings{petersen2022style,
  author = {Felix Petersen and Bastian Goldluecke and Oliver Deussen and Hilde Kuehne},
  title = {{Style Agnostic 3D Reconstruction via Adversarial Style Transfer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00233},
}
Figure