OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

Mariia Gladkova, Neehar Peri, Ishan Khatri, Deva Ramanan, Daniel CremersarXiv 2026 · arXiv

TL;DR

OV3D-Bench exposes a critical flaw in how we evaluate open-vocabulary 3D object detectors: current "target-aware" protocols use unrealistic per-image oracles that hide massive semantic failures. By introducing a realistic dataset-level benchmark that decouples geometry from semantics, the authors reveal that while models can accurately locate objects in 3D space, they severely struggle with fine-grained classification and prompt phrasing.

What Problem Does This Paper Address?

Open-vocabulary monocular 3D object detection is crucial for fields like autonomous driving and robotics, allowing models to detect arbitrary objects from a single image using natural language queries. However, current evaluation practices paint an overly optimistic picture of model performance.

Most existing benchmarks use a "target-aware" protocol, meaning the model is only prompted with the specific categories known to be present in the test image. In the real world, a deployed system doesn't have a magical oracle telling it what's in the scene beforehand. This protocol silently suppresses hallucinations and false positives. Furthermore, standard evaluation metrics collapse localization quality (geometry) and classification correctness (semantics) into a single Average Precision ($mAP_{3D}$) score, making it impossible to diagnose exactly why a model is failing.

Methods

To fix these evaluation blind spots, the authors introduce OV3D-Bench, which implements two key changes:

  1. Dataset-Level Prompting: Instead of per-image oracles, models are prompted with the dataset's entire vocabulary at test time. This removes privileged information while preserving realistic scene-level priors (e.g., an urban driving dataset contains cars, not sofas).
  2. Decoupled Evaluation: The benchmark splits performance into three distinct axes: geometric accuracy (using 3D Class-Agnostic Recall), semantic robustness (analyzing confusion matrices and prompt phrasing sensitivity), and cross-domain generalization.

Additionally, the authors propose a surprisingly simple Training-Free VLM Remapping Baseline. Instead of training a complex open-vocabulary architecture from scratch, they take a frozen, closed-vocabulary 3D detector (like Cube R-CNN), project its 3D bounding boxes onto 2D image crops, and use a powerful vision-language model (SigLIPv2) to remap the closed-world predictions to open-vocabulary labels.

Fig. 1: Open-Vocabulary Monocular 3D Detectors Struggle with Vocab- ulary, Not Geometry. All five detectors consistently localize a regular vehicle (mean 3D IoU 0.58, BEV insets) yet disagree on its category (top). Prompt phrasing swings accuracy sharply for prompt-conditioned detectors but far less for our SigLIPv2- based remapping baseline (bottom left). The target-aware protocol hides these semantic errors, inflating scores most where the dataset-level score is lowest. (bottom right).

Datasets

OV3D-Bench evaluates models across seven diverse indoor and outdoor datasets to stress-test cross-domain generalization:

  • In-domain (Omni3D taxonomy): KITTI, nuScenes, SUNRGBD, ARKitScenes, and Hypersim.
  • Zero-shot/Out-of-domain: Argoverse 2 (AV2) and ScanNet-200 (featuring fine-grained categories like "hair dryer" and "soap dispenser").

Key Insights and Ideas

  • Geometry is Mature, Semantics is the Bottleneck: Open-vocabulary detectors are actually quite good at finding objects in 3D space. As shown in Figure 1, multiple models consistently localize the same vehicle with high 3D IoU, but they completely disagree on what it is—confusing a regular car with a truck, truck cab, or railed vehicle.
  • The "Oracle" Illusion: Target-aware evaluation severely inflates performance. When you force a model to evaluate against the entire dataset vocabulary, its apparent accuracy plummets because it can no longer hide its cross-class false positives and hallucinations.
  • Extreme Prompt Sensitivity: Purpose-built open-vocabulary models are incredibly fragile when it comes to how a query is phrased. Adding descriptive attributes can completely break their detection capabilities, whereas the authors' simple VLM remapping approach remains highly stable.

Key Results

  • Metric Inflation: The target-aware protocol inflates DetAny3D’s performance by 1.1$\times$ on ARKitScenes and a massive 1.9$\times$ on ScanNet-200 compared to the realistic dataset-level protocol.
  • Prompt Collapse: When changing a prompt from "car" (underdetailed) to "a detailed high-resolution photo of a car" (overdetailed), WildDet3D’s performance collapses from 18.6 to 5.4 $AP_{3D}$ on KITTI. Meanwhile, the SigLIPv2 remapping baseline stays within a stable 1.7 $AP_{3D}$ margin.
  • Baseline Superiority: The simple, training-free SigLIPv2 remapping baseline matches or outperforms recent purpose-built open-vocabulary methods on unseen outdoor domains.
  • Efficiency: The remapping baseline (Cube R-CNN*) is highly efficient, running at 223.9 ms/image—making it 2.6$\times$ faster than DetAny3D because it avoids heavy SAM-based 3D heads.

Limitations and Future Work

While the VLM remapping baseline serves as an excellent diagnostic probe, it is not yet a perfect deployable detector. It struggles to cleanly separate true positives from false positives because the contrastive VLM's cosine similarities frequently overlap. The authors suggest that lightweight calibration (such as training a small MLP on held-out similarity scores) could help resolve this. Furthermore, the benchmark inherits the non-exhaustive annotations of its underlying datasets, meaning correct detections of unannotated objects are still penalized as false positives.

Summary updated

Abstract

Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image category oracles unavailable at deployment, and all collapse geometry and semantics into a single AP metric. To address this, we introduce OV3D-Bench, a diagnostic benchmark that compares open-vocabulary monocular 3D detectors under deployment-realistic conditions across seven indoor and outdoor datasets. Our benchmark replaces the per-image class name oracle with test-time dataset-level class name prompts, and decouples detection accuracy along three axes: localization, semantic robustness, and cross-domain transfer. We evaluate seven representative detectors and find that (i) they localize objects well yet often mislabel a correctly localized box as a semantically adjacent category; (ii) accuracy is highly sensitive to prompt phrasing (e.g. WildDet3D's performance collapses from 18.6 to 5.4 AP when prompted with "a detailed high-resolution photo of a car" rather than "car"); and (iii) the widely adopted target-aware protocol hides these errors (e.g. inflating DetAny3D's AP by 1.9 $\times$ on ScanNet). Lastly, we demonstrate that simply remapping a frozen closed-vocabulary detector's predictions using a contrastive vision-language encoder such as SigLIPv2 performs competitively against recent purpose-built open-vocabulary methods. This indicates that geometric localization is more mature, while open-vocabulary semantics remains the primary bottleneck.
BibTeX
@misc{gladkova2026ov,
  author = {Mariia Gladkova and Neehar Peri and Ishan Khatri and Deva Ramanan and Daniel Cremers},
  title = {{OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection}},
  year = {2026},
  eprint = {2608.17110},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.17110},
}
Figure