GATOR: Generative and Agentic 3D Object Reconstruction From Casual Images

Qirui Wu, Stan Birchfield, Hesam Rabeti, Angel X. Chang, Bowen WenarXiv 2026 · arXiv

TL;DR

GATOR is a new framework that combines generative 3D modeling with agentic refinement to reconstruct high-fidelity, scene-aligned 3D objects from just a few "casual" (uncalibrated and cluttered) images. By using a generative model for a "warm start" and a GPT-6-powered agent for targeted structural and texture repairs, GATOR produces simulation-ready assets that significantly outperform both pure generative baselines and agents building from scratch.

What Problem Does This Paper Address?

Creating high-quality 3D assets from everyday photographs is a "holy grail" for simulation, gaming, and AR. However, "casual" captures—photos taken without professional rigs, calibrated cameras, or isolated backgrounds—present massive hurdles.

Current methods generally fall into two camps:

  1. Generative Models: These are great at hallucinating missing parts of an object but often produce "mushy" geometry, incorrect topology (like fused whisk wires), or garbled textures.
  2. Agentic Models: Multimodal agents (like GPT-6) can use graphics tools like Blender to build precise geometry, but building a complex asset from scratch is incredibly slow, expensive, and often results in oversimplified shapes.

Furthermore, most methods struggle to place the reconstructed object back into its original scene with the correct pose and scale, especially when the object is partially occluded by clutter.

Methods

GATOR (Generative and Agentic 3D Object Reconstruction) tackles these issues through a two-stage pipeline:

1. Pose-Aware Generative Reconstruction

GATOR starts with a generative cascade (structure $\rightarrow$ geometry $\rightarrow$ appearance). The key innovation here is the Local Modality Mixer. Instead of feeding RGB, masks, and depth maps into the model separately, GATOR fuses them at the patch level. This allows the model to "cross-check" modalities—for instance, using the mask to ignore background clutter while using the RGB and pointmap to understand the object's 3D structure.

It also introduces stage-specific text guidance. Category names guide the initial structure, while detailed captions (generated via VLM) guide the fine geometry and PBR (Physically Based Rendering) material generation.

The GATOR architecture showing the generative flow and agentic refinement loop.

2. Agentic Refinement

The generated asset is often 90% there but might have "fused" openings or blurry logos. GATOR hands this asset to a multimodal agent (GPT-6-Astra). The agent doesn't start from scratch; it uses the generated mesh as a "scaffold." It inspects the original images, renders the current mesh, and writes Blender scripts to:

  • Complete missing parts (e.g., chair legs hidden by a table).
  • Repair topology (e.g., turning a solid block into a wire mesh strainer).
  • Refine textures and UV maps (e.g., fixing garbled text).

The agent operates in an edit-render-review loop, reverting any changes that decrease the fidelity to the original observations.

Datasets

The authors trained GATOR on S3 (Scalable Synthetic Scenes), a massive new corpus of 980,000 assets placed in procedurally cluttered synthetic environments to mimic the messiness of real-world photos.

For evaluation, they tested across a diverse range of benchmarks:

  • Toys4K: A synthetic dataset of 105 categories.
  • LM-O, HB, and HANDAL: Real-world tabletop datasets featuring cluttered environments.
  • ScanNet++: Real-world indoor room scans, testing the model's ability to handle large-scale scene context.

Key Insights and Ideas

  • Generative Initialization is a Game-Changer: The most striking finding is that giving an agent a "warm start" is vastly more efficient than building from scratch. GATOR's generative stage takes ~30 seconds; an agent starting from that point outperforms an agent working for 20 minutes from a blank canvas.
  • Local Fusion vs. Global Reasoning: By mixing modalities (RGB, mask, pointmap) locally at the patch level before global cross-view attention, the model becomes much more robust to noisy inputs like imprecise masks or depth estimates.
  • Simulation Readiness: Purely generative models often produce "unstable" objects. The agentic refinement specifically fixes structural issues (like slanted legs), making the resulting assets stable in physics simulators like PyBullet.
Qualitative results showing how the agent fixes irregular holes and missing parts.

Key Results

  • Quantitative Superiority: On the Toys4K benchmark, GATOR reduced Chamfer Distance (a measure of geometric error) by 19.7% compared to the strongest baseline (Pixal3D-MV).
  • Real-World Performance: On tabletop datasets, GATOR roughly halved the geometric error of previous state-of-the-art methods while significantly improving texture quality (PSNR).
  • Efficiency: As shown in Figure 6, GATOR achieves higher quality in under a minute than GPT-6-Astra can achieve in 20 minutes when starting from scratch.
Reconstruction quality vs. time, showing GATOR's efficiency advantage.

Limitations and Future Work

While GATOR is robust, it still relies on external models for initial mask and depth estimates. A "catastrophic" failure in the initial segmentation can still derail the reconstruction. Additionally, the agentic refinement currently relies on a proprietary model (GPT-6-Astra) and can take up to 10 minutes per object for full refinement.

The authors suggest that future work should focus on tighter integration between the segmentation/geometry estimators and the reconstruction model, as well as exploring open-source agent alternatives.

Summary updated

Abstract

Reconstructing complete, scene-aligned 3D objects from casual images requires integrating sparse, uncertain observations and inferring surfaces hidden by occlusions. We present GATOR, a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images. Our local modality mixer couples patch-aligned RGB, target-mask, and pointmap features before cross-view reasoning, preserving scene context while distinguishing the target from its surroundings. Text-guided semantic conditioning complements these spatial cues with category names and object captions through stage-specific adapters for structure, geometry, and appearance generation. The generated asset initializes a multimodal agent, providing instance-specific geometry and pose for targeted structural and texture refinement through an observation-guided edit-render-review loop. Across synthetic objects, cluttered tabletops, and indoor scenes, GATOR achieves strong geometric and appearance fidelity while recovering scene-relative pose from sparse observations. Time-budget comparisons and scene-level simulation further demonstrate the reconstruction efficiency and simulation readiness. Project page: https://research.nvidia.com/labs/lpr/gator/
BibTeX
@misc{wu2026gator,
  author = {Qirui Wu and Stan Birchfield and Hesam Rabeti and Angel X. Chang and Bowen Wen},
  title = {{GATOR: Generative and Agentic 3D Object Reconstruction From Casual Images}},
  year = {2026},
  eprint = {2610.11215},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.11215},
}
Figure