Abstract
ObjVariantEnsemble (OVE) is a novel framework designed to stress-test 3D Large Language Models (LLMs) by placing them in environments containing subtly distinguished objects. By combining diverse object-level datasets with an automated annotation pipeline leveraging 2D Vision-Language Models (VLMs) and LLMs, OVE generates a large-scale benchmark of 75,000 scene-text pairs. Evaluation of state-of-the-art models reveals significant weaknesses in fine-grained shape recognition and pure spatial reasoning, providing a new frontier for 3D scene understanding research.
Introduction
While 3D scene understanding has progressed rapidly, existing benchmarks like ScanNet or ScanRefer have reached a performance plateau. These datasets often suffer from small scales, vague annotations, and a lack of customizability. A critical gap remains in fine-grained discrimination: if a prompt refers to "the brown chair" in a room with multiple similar chairs, current models often fail because the linguistic cues are insufficient to distinguish the target from its neighbors. OVE addresses these gaps by creating complex "kitbashed" scenes where targets are surrounded by distractors that share the same class or color but differ in subtle geometric details.
Methodology
1. Scene Construction (The Ensemble)
Instead of relying on expensive new 3D scans, OVE populates existing scene backgrounds (e.g., from ScanNet) with high-quality objects from datasets like OmniObject or ShapeNet.
- Object Retriever: This module identifies "distractors"—objects that are semantically similar to the target (same class/color) but geometrically distinct.
- Scene Integrator: It places these objects into the 3D scene, ensuring realistic spatial relationships (e.g., "between," "to the left of") that are mathematically verified for ground-truth accuracy.
2. Automated Annotation (The Annotator)
To scale descriptive annotations without human labor, OVE utilizes a multi-stage pipeline involving 2D VLMs and LLMs:
- Multi-View Rendering: The 3D target and distractors are rendered from multiple 2D angles to capture geometric nuances.
- Iterative QA: An LLM (e.g., GPT-4) prompts a VLM (e.g., LLaVA) to identify specific differences in features like seat shape, backrest style, or height.
- Summarization: The LLM distills these observations into highly descriptive prompts, such as: "The chair features a straight design with a high back and a slatted backrest. It's between two chairs."
The OVE Benchmark
The resulting OVE Benchmark comprises 75,000 scene-text pairs, drawing from object sources like ModelNet and OmniObject, and scene sources like Multi3DRef. This scale and semantic complexity far exceed previous 3D grounding datasets, offering a more rigorous test of model intelligence.
Key Insights and Results
Evaluations of state-of-the-art models, including Chat-3D-v2 and M3DRef-CLIP, yielded several critical findings:
- Spatial Reasoning Bottleneck: Model accuracy drops precipitously when prompts rely solely on spatial relationships (e.g., "the chair on the right") without class or color cues. This suggests that current 3D LLMs rely heavily on object recognition rather than true spatial understanding.
- The Distractor Effect: Performance decreases linearly as the number of similar distractors increases, quantifying the impact of scene clutter on model reliability.
- Shape vs. Color: Models generally perform better at distinguishing objects by shape or class than by pure location or color. While fine-tuning on OVE data improves performance (e.g., from ~25% to ~60% on shape-based tasks), accuracy on pure location tasks remains low (~19%).
Conclusion and Future Work
OVE demonstrates that current 3D LLMs struggle with the subtle geometric and spatial nuances required for advanced scene understanding. While the framework's reliance on 2D renders and synthetic scene integration presents some limitations regarding naturalism, it provides a scalable method for identifying model failure points. Future work will focus on expanding spatial predicates (e.g., "underneath," "hidden") and utilizing generative models to create even more realistic 3D object variations.
Summary updated