TL;DR
GATOR is a new framework that combines generative 3D modeling with agentic refinement to reconstruct high-fidelity, scene-aligned 3D objects from just a few "casual" (uncalibrated and cluttered) images. By using a generative model for a "warm start" and a GPT-6-powered agent for targeted structural and texture repairs, GATOR produces simulation-ready assets that significantly outperform both pure generative baselines and agents building from scratch.
What Problem Does This Paper Address?
Creating high-quality 3D assets from everyday photographs is a "holy grail" for simulation, gaming, and AR. However, "casual" captures—photos taken without professional rigs, calibrated cameras, or isolated backgrounds—present massive hurdles.
Current methods generally fall into two camps:
- Generative Models: These are great at hallucinating missing parts of an object but often produce "mushy" geometry, incorrect topology (like fused whisk wires), or garbled textures.
- Agentic Models: Multimodal agents (like GPT-6) can use graphics tools like Blender to build precise geometry, but building a complex asset from scratch is incredibly slow, expensive, and often results in oversimplified shapes.
Furthermore, most methods struggle to place the reconstructed object back into its original scene with the correct pose and scale, especially when the object is partially occluded by clutter.
Methods
GATOR (Generative and Agentic 3D Object Reconstruction) tackles these issues through a two-stage pipeline:
1. Pose-Aware Generative Reconstruction
GATOR starts with a generative cascade (structure $\rightarrow$ geometry $\rightarrow$ appearance). The key innovation here is the Local Modality Mixer. Instead of feeding RGB, masks, and depth maps into the model separately, GATOR fuses them at the patch level. This allows the model to "cross-check" modalities—for instance, using the mask to ignore background clutter while using the RGB and pointmap to understand the object's 3D structure.
It also introduces stage-specific text guidance. Category names guide the initial structure, while detailed captions (generated via VLM) guide the fine geometry and PBR (Physically Based Rendering) material generation.
2. Agentic Refinement
The generated asset is often 90% there but might have "fused" openings or blurry logos. GATOR hands this asset to a multimodal agent (GPT-6-Astra). The agent doesn't start from scratch; it uses the generated mesh as a "scaffold." It inspects the original images, renders the current mesh, and writes Blender scripts to:
- Complete missing parts (e.g., chair legs hidden by a table).
- Repair topology (e.g., turning a solid block into a wire mesh strainer).
- Refine textures and UV maps (e.g., fixing garbled text).
The agent operates in an edit-render-review loop, reverting any changes that decrease the fidelity to the original observations.
Datasets
The authors trained GATOR on S3 (Scalable Synthetic Scenes), a massive new corpus of 980,000 assets placed in procedurally cluttered synthetic environments to mimic the messiness of real-world photos.
For evaluation, they tested across a diverse range of benchmarks:
- Toys4K: A synthetic dataset of 105 categories.
- LM-O, HB, and HANDAL: Real-world tabletop datasets featuring cluttered environments.
- ScanNet++: Real-world indoor room scans, testing the model's ability to handle large-scale scene context.
Key Insights and Ideas
- Generative Initialization is a Game-Changer: The most striking finding is that giving an agent a "warm start" is vastly more efficient than building from scratch. GATOR's generative stage takes ~30 seconds; an agent starting from that point outperforms an agent working for 20 minutes from a blank canvas.
- Local Fusion vs. Global Reasoning: By mixing modalities (RGB, mask, pointmap) locally at the patch level before global cross-view attention, the model becomes much more robust to noisy inputs like imprecise masks or depth estimates.
- Simulation Readiness: Purely generative models often produce "unstable" objects. The agentic refinement specifically fixes structural issues (like slanted legs), making the resulting assets stable in physics simulators like PyBullet.
Key Results
- Quantitative Superiority: On the Toys4K benchmark, GATOR reduced Chamfer Distance (a measure of geometric error) by 19.7% compared to the strongest baseline (Pixal3D-MV).
- Real-World Performance: On tabletop datasets, GATOR roughly halved the geometric error of previous state-of-the-art methods while significantly improving texture quality (PSNR).
- Efficiency: As shown in Figure 6, GATOR achieves higher quality in under a minute than GPT-6-Astra can achieve in 20 minutes when starting from scratch.
Limitations and Future Work
While GATOR is robust, it still relies on external models for initial mask and depth estimates. A "catastrophic" failure in the initial segmentation can still derail the reconstruction. Additionally, the agentic refinement currently relies on a proprietary model (GPT-6-Astra) and can take up to 10 minutes per object for full refinement.
The authors suggest that future work should focus on tighter integration between the segmentation/geometry estimators and the reconstruction model, as well as exploring open-source agent alternatives.
Summary updated