Infer3D: Inverse Generative Modeling Enables Robust 3D Reconstruction

Amani Kiruga, Mark Hamilton, Joshua B. Tenenbaum, William T. Freeman, Vincent Sitzmann, Yilun Du, Ayush TewariarXiv 2026 · arXiv

TL;DR

Infer3D moves beyond brittle feedforward 3D reconstruction by adopting a "top-down" generative approach inspired by human vision. By optimizing the latent variables of a 2D generative prior coupled with a 3D lifting network at test time, it achieves remarkable robustness to extreme viewpoints, sensor distortions, and unfamiliar textures that cause standard models to collapse.

What Problem Does This Paper Address?

Modern monocular 3D reconstruction has largely become a "pattern matching" game. State-of-the-art models typically use amortized, feedforward architectures—essentially encoders that map a 2D image directly to a 3D representation like Gaussians or NeRFs. While incredibly fast, these models are notoriously brittle. If you show them an object from an unconventional angle, under strange lighting, or through a distorted lens (out-of-distribution or OOD conditions), they often produce "flat" or nonsensical geometries.

The authors argue that this happens because these models lack an internal "mental model" of the physical world. Humans, by contrast, use a generative inference strategy: we maintain internal hypotheses about 3D structure and continuously match them against sensory input. Infer3D aims to bridge this gap by formalizing 3D reconstruction as an inverse generative problem.

Standard feedforward models fail rapidly when camera elevation or roll deviates from the training distribution, whereas Infer3D maintains coherent structure.

Methods

The core of Infer3D is an analysis-by-synthesis framework. Instead of a direct mapping, it searches for the scene geometry and rendering parameters that best explain the observed image.

The Architecture

The pipeline consists of three main components:

  1. Generative Image Prior ($G$): A pre-trained 2D generator (like StyleGAN2 or a Diffusion Autoencoder) that defines a manifold of valid, "in-distribution" images.
  2. 3D Lifting Network ($\Phi$): A frozen network (e.g., Splatter Image) that takes a 2D image and "lifts" it into a 3D representation (3D Gaussians).
  3. Differentiable Renderer ($R$): A standard renderer that produces a 2D image from the 3D Gaussians given specific camera parameters.

The Optimization Loop

To reconstruct an OOD image, Infer3D freezes the networks and optimizes the Latent Code ($z$) and Rendering Parameters ($\theta$). The goal is to minimize the reconstruction loss between the rendered output and the input image. Crucially, because the 3D lifter is frozen, the optimization is constrained to produce a valid, multi-view-consistent 3D manifold rather than just a "flat" image that happens to match the pixels.

The Infer3D inference paradigm uses a test-time optimization loop to align a generative prior with OOD observations.

Adaptive Inference

Recognizing that optimization is slow, the authors introduced an adaptive strategy. They use the initial reconstruction error of the generative prior as an OOD signal. If the error is low, the system trusts the fast feedforward pass. If the error is high (indicating a confusing or OOD stimulus), it triggers the full optimization loop.

Datasets

The authors evaluated Infer3D across several challenging benchmarks:

  • ShapeNet: Used for controlled experiments on extreme viewpoint shifts (SO(3) rotations and SE(3) translations).
  • CO3D (Common Objects in 3D): Specifically the Hydrants and Vases categories, used to test generalization to real-world object categories.
  • RealEstate10K (RE10K): Used to evaluate robustness to non-linear sensor distortions (fisheye lenses).
  • RealCars: A dataset of "in-the-wild" car captures with complex reflections and lighting to test appearance robustness.

Key Insights and Ideas

  • The "3D Manifold" Constraint: The most clever part of the paper is using a frozen 3D lifter. If you just optimized a 2D generator to match an image, it would likely ignore 3D consistency. By forcing the output through a 3D lifter, the latent space $z$ effectively becomes a proxy for valid 3D shapes.
  • OOD Detection for Free: In most systems, OOD detection requires a separate classifier. Here, it's a natural byproduct of the generative prior. If the model can't easily "imagine" the input image using its learned prior, it knows it needs to work harder (optimize).
  • Disentangling Sensor from Scene: By including camera intrinsics and distortion parameters in the optimization ($\theta$), Infer3D can "unwarp" fisheye distortions, separating the lens's effects from the actual 3D geometry of the scene.

Key Results

Infer3D significantly outperforms feedforward baselines in OOD scenarios:

  • Viewpoint Robustness: On ShapeNet cars with extreme rotations, Infer3D achieved a PSNR of 20.29 dB, compared to just 14.30 dB for the BRFI baseline.
  • Sensor Shifts: When faced with fisheye distortion, standard models (like CATSplat) interpret lens warping as curved geometry. Infer3D successfully disentangles the two, recovering clean 3D structures (see Figure 4).
  • Real-World Generalization: In "in-the-wild" tests on smartphone photos, Infer3D remained robust to cluttered backgrounds and partial occlusions where feedforward models failed catastrophically.
Qualitative results showing Infer3D's robustness to viewpoint shifts (a) and sensor distortions (b) compared to baselines.

Limitations and Future Work

The primary drawback is computational cost. While the adaptive strategy helps, the full optimization loop takes roughly 2.8 minutes per OOD sample—orders of magnitude slower than a millisecond-scale feedforward pass.

Additionally, the visual sharpness of the reconstruction is limited by the underlying 2D generative prior. The authors suggest that integrating more powerful, state-of-the-art diffusion models as the generative backbone could improve high-frequency detail, albeit at an even higher computational cost. Future work may also explore more efficient optimization techniques to bring "analysis-by-synthesis" closer to real-time performance.

Summary updated

Abstract

Humans can understand the 3D structure of scenes even from highly unfamiliar or extreme viewpoints. Cognitive science suggests this generalization stems from a top-down generative inference strategy. In contrast, state-of-the-art 3D reconstruction neural networks typically rely on amortized, feedforward inference. While fast, this pattern-matching approach degrades when faced with out-of-distribution (OOD) scene conditions unseen during training. Inspired by the generative nature of human vision, we propose Infer3D, an inverse generative modeling framework that recovers the latent variables of a generative model to best match the monocular input. To prevent degenerate flat outputs, we couple the generative image prior with a frozen 3D lifting network that constrains the optimization space to a valid, multi-view-consistent 3D manifold. Across a variety of controlled synthetic and real-world tasks, Infer3D demonstrates superior OOD reconstruction accuracy compared to feedforward networks trained on equal or larger datasets. Furthermore, mirroring how human perception requires more time to process confusing stimuli, we introduce an adaptive inference strategy that expends more test-time compute on ambiguous OOD examples while efficiently routing in-distribution inputs. Project website: https://amanikiruga.github.io/infer3d-site/
BibTeX
@misc{kiruga2026infer,
  author = {Amani Kiruga and Mark Hamilton and Joshua B. Tenenbaum and William T. Freeman and Vincent Sitzmann and Yilun Du and Ayush Tewari},
  title = {{Infer3D: Inverse Generative Modeling Enables Robust 3D Reconstruction}},
  year = {2026},
  eprint = {2611.25061},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25061},
}
Figure