Abstract
Open-set object detection (OSOD) localizes objects while identifying and rejecting unknown classes at inference. While recent OSOD models perform well on benchmarks, their behavior under realistic user prompting remains underexplored. In interactive XR settings, user-generated prompts are often ambiguous, underspecified, or overly detailed. To study prompt-conditioned robustness, we evaluate two OSOD models, GroundingDINO and YOLO-E, on real-world XR images and simulate diverse user prompting behaviors using vision-language models. We consider four prompt types: standard, underdetailed, overdetailed, and pragmatically ambiguous, and examine the impact of two enhancement strategies on these prompts. Results show that both models exhibit stable performance under underdetailed and standard prompts, while they suffer degradation under ambiguous prompts. Overdetailed prompts primarily affect GroundingDINO. Prompt enhancement substantially improves robustness under ambiguity, yielding gains exceeding 55% mIoU and 41% average confidence. Based on the findings, we propose several prompting strategies and prompt enhancement methods for OSOD models in XR environments.
Introduction: The Challenge of Real-World XR Prompting
Open-Set Object Detection (OSOD) represents a significant leap forward for Extended Reality (XR) systems, allowing them to locate objects based on free-form text descriptions rather than a fixed list of predefined classes. This flexibility is essential for immersive environments where users may wish to interact with any arbitrary object in their physical space.
However, a critical gap exists between laboratory benchmarks and real-world application. Current OSOD benchmarks typically assume clean, precise text prompts (e.g., "a red coffee mug"). In actual XR usage, human communication is inherently messy. Users are frequently vague ("I'm thirsty"), overly specific ("a glossy red cylindrical shaker with a rounded top"), or under-detailed. This research addresses prompt-conditioned robustness: the ability of state-of-the-art OSOD models to handle realistic, flawed human instructions and the potential for Vision-Language Models (VLMs) to mitigate these failures.
Methodology: Simulating the Messy User
To evaluate these systems at scale without the prohibitive cost of human trials, an automated evaluation pipeline was developed. The process involves three main components:
- User Simulation: A Vision-Language Model (GPT-5) was employed to generate four distinct prompt types based on target objects in AR images: underdetailed, standard, overdetailed, and pragmatically ambiguous (e.g., describing a mouse as "the device I slide around to move the cursor").
- Model Evaluation: These prompts were tested against two leading OSOD architectures: GroundingDINO (GD) and YOLO-E.
- Prompt Enhancement: A VLM-based "middleman" was introduced to intercept and refine user prompts before they reach the detection model. Two strategies were tested: Key Object Extraction (distilling the prompt to essential noun phrases) and Semantic Category Grounding (mapping ambiguous intent to standard taxonomy categories).
Datasets and Evaluation
The study utilized a curated set of 264 images from the DiverseAR and DiverseAR+ datasets. These images capture authentic Augmented Reality scenarios from headsets and mobile devices, characterized by cluttered backgrounds, varying viewpoints, and virtual overlays. Ground truth bounding boxes were manually annotated to provide a rigorous baseline for testing prompt variations.
Key Insights: Why Models Fail
The evaluation revealed several counterintuitive findings regarding model behavior:
- The Over-Specification Paradox: More detail is not always beneficial. Over-specifying an object with numerous attributes often confuses models like GroundingDINO, as the system struggles to satisfy too many competing visual constraints simultaneously.
- The Ambiguity Barrier: "Pragmatic ambiguity" is a primary failure mode. Current OSOD models lack the reasoning capabilities to map implicit intent (e.g., "I'm thirsty") to a spatial prediction (e.g., a water bottle).
- Architectural Differences: YOLO-E, which utilizes a recall-oriented strategy with many candidate boxes, proved more resilient to overdetailed prompts than GroundingDINO, though both models failed significantly on ambiguous inputs.
Results: The Power of Prompt Enhancement
While baseline performance was stable for standard and underdetailed prompts (GroundingDINO mIoU ~86%, YOLO-E mIoU ~63-65%), performance plummeted on ambiguous prompts. GroundingDINO’s mIoU dropped to 35.84%, while YOLO-E’s fell to 9.56%.
However, the introduction of VLM-based enhancement led to massive performance recoveries. By using Semantic Category Grounding to translate human intent into standard object categories, GroundingDINO’s mIoU recovered by 55.15% (reaching 90.99%), and YOLO-E’s jumped by 56.10% (reaching 65.66%). Furthermore, Key Object Extraction boosted detection confidence scores for GroundingDINO by over 41%.
Conclusion
This study demonstrates that while OSOD models are powerful, they require a "semantic translator" to bridge the gap between human intent and visual detection. By utilizing VLMs as a pre-processing layer, XR systems can handle the cognitive heavy lifting of interpreting ambiguous user prompts, allowing detection models to focus on their core strength: spatial localization. Future work will expand this research to continuous video streams and live human-in-the-loop testing to further refine these prompting strategies.
Summary updated