Ultimate3D: Teaching Multimodal LLMs 3D Camera-Object Relations by Large-Scale Visual Instruction Data

Liu He, Yizhi Song, Ruiqi Xu, Baris Sarper Tezcan, Xiao Zeng, Albert Y. C. Chen, Lu Xia, Shashwat Verma, Sankalp Dayal, Min Sun, Cheng-Hao Kuo, Daniel AliagaarXiv 2026 · arXiv

TL;DR

Ultimate3D introduces a novel synthetic data pipeline designed to cure the "spatial blindness" of Multimodal Large Language Models (MLLMs). By combining 3D asset rendering with diffusion-based image generation and LLM-driven instruction tuning, the authors created a massive dataset of 240,000 VQA pairs that teach models precise camera-object relations. The resulting fine-tuned open-source models significantly outperform commercial heavyweights like GPT-5.5 and Gemini-3.5 on 3D spatial reasoning tasks.

What Problem Does This Paper Address?

Despite their impressive capabilities, current MLLMs are surprisingly poor at understanding 3D spatial relationships. If you ask a state-of-the-art model whether a car in a photo is facing "front-left" or "back-right," or whether a shot is a "close-up" versus a "medium-shot," it often fails.

The authors identify two primary causes for this:

  1. Data Bias: Most internet-scraped images are captured from canonical, front-facing viewpoints. Models learn this bias and default to "front" or "top" views when uncertain.
  2. Lack of Ground Truth: Existing datasets rarely include precise 3D annotations for camera elevation, azimuth, or distance.

Ultimate3D aims to bridge this gap by providing a scalable way to generate photorealistic training data where the 3D geometry is known with mathematical precision.

Methods

The core of the paper is a synthetic generation pipeline that transforms 3D assets into high-quality visual instruction data. The process follows four main steps:

  1. 3D Visual Prior Rendering: Using a 3D renderer (Blender), the system takes a 3D asset and renders "priors"—specifically depth maps and Canny edges—from a precisely defined camera pose $\beta$ (defined by azimuth $\phi$, elevation $\theta$, and distance $D$).
  2. Image Description Generation: An LLM (GPT-4o) generates a diverse text prompt describing the object and a realistic background context.
  3. Controlled Image Generation: These 3D priors and text prompts are fed into a Multi-ControlNet framework with an SDXL backbone. This ensures the final image is photorealistic and diverse while strictly adhering to the 3D geometry of the original asset.
  4. Text Instruction Generation: Finally, GPT-4o generates multiple-choice Visual Question Answering (VQA) pairs based on the known ground-truth camera-object relations.
Fig. 2: Our Framework. Each shade box corresponds to a section in Sec. 3. Given open-sourced 3D assets, our approach leverages a 3D Renderer to generate 3D visual priors (Iβ) preserving ground truth camera-object relation (β). Meanwhile, LLMs take 3D asset category to generate diverse image descriptions (Timg) as conditional guidance. Both the 3D visual priors and diverse text prompts are used to generate synthetic images (Isyn) by multiple ControlNet-based networks. Corresponding text QA instructions (Tqa) are generated by LLMs given the ground truth camera-object relation. Our Ultimate3D dataset and benchmark (i.e. pairs of Tqa and Isyn) contribute to fine-tuning and evaluation of MLLMs.

Datasets

The authors introduce Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset:

  • Training Set: 240,000 VQAs derived from 85,000 synthetic images (using 1,180 3D assets across 100 categories) and 18,000 real human crops from the MEBOW dataset.
  • Benchmark: A rigorous 8,000-sample benchmark consisting of a Synthetic Set (manually reviewed for correctness) and a Real Set (using images from Pascal3D+ and MEBOW) to test generalization to real-world photography.

Key Insights and Ideas

  • Geometry as an Anchor: The researchers found that text prompts alone are insufficient to control 3D orientation in diffusion models. Using 3D priors (depth/Canny) is essential to "lock" the geometry, preventing the model from hallucinating a more common "front-facing" view.
  • Synthetic-to-Real Transfer: A major "aha" moment is that models trained on these synthetic, rendered images generalize remarkably well to real-world photos. This suggests that "camera-object relation" is a fundamental spatial concept that MLLMs can learn regardless of whether the pixels are real or generated.
  • The "Thinking" Limit: Interestingly, increasing the "thinking" budget of commercial models (like Gemini-3.5-Flash) improved orientation accuracy but actually decreased camera-shot accuracy. This highlights that spatial reasoning is a data-bottleneck problem, not just a compute-bottleneck one.

Key Results

The quantitative improvements are stark. Models fine-tuned on Ultimate3D achieved an average accuracy improvement of 29.7% across camera-object recognition tasks.

  • LLaVA-1.6-13B: Orientation accuracy soared from a baseline of 16.1% to 72.4%.
  • Qwen3-VL-32B: Achieved the strongest overall performance among open models, reaching 75.7% average accuracy.
  • Outperforming SOTA: Fine-tuned open-source models (Gemma-4, Qwen3) consistently beat closed-source commercial models like GPT-5.5 and Gemini-3.5 on the Ultimate3D benchmark.
Qualitative results comparing commercial models and the Ultimate3D-finetuned Gemma model.

Limitations and Future Work

While Ultimate3D is a major step forward, the authors note several areas for growth:

  • Numerical Precision: Models still struggle with predicting exact numerical degrees (e.g., "facing 136.5 degrees") compared to categorical bins (e.g., "facing front-left").
  • Scene Complexity: The current dataset focuses on single-object scenes. Future work will need to address multi-object relations and more complex environmental interactions.
  • Real-World Clutter: Synthetic "long shots" are often cleaner than real-world wide shots, leading to a slight performance gap when evaluating on cluttered real-world benchmarks.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset. Ultimate3D consists of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 29.7% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.
BibTeX
@misc{he2026ultimate,
  author = {Liu He and Yizhi Song and Ruiqi Xu and Baris Sarper Tezcan and Xiao Zeng and Albert Y. C. Chen and Lu Xia and Shashwat Verma and Sankalp Dayal and Min Sun and Cheng-Hao Kuo and Daniel Aliaga},
  title = {{Ultimate3D: Teaching Multimodal LLMs 3D Camera-Object Relations by Large-Scale Visual Instruction Data}},
  year = {2026},
  eprint = {2611.24795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24795},
}
Figure