TL;DR
This paper investigates how "messy" human language—ranging from vague hints to overly wordy descriptions—degrades the performance of open-set object detection (OSOD) in Augmented Reality. By using Vision-Language Models (VLMs) to "translate" these realistic user prompts into concise, model-friendly instructions, the researchers achieved massive performance gains, including a 55% improvement in localization accuracy (mIoU) for ambiguous queries.
What Problem Does This Paper Address?
Open-set object detection (OSOD) is a breakthrough in computer vision because it allows a model to find objects it wasn't specifically trained on, guided simply by a text prompt (e.g., "find the plush dinosaur"). While these models excel on clean, academic benchmarks, the authors argue that real-world Extended Reality (XR) users don't talk like benchmarks.
In a real AR setting, a user might be:
- Underdetailed: "The bottle."
- Overdetailed: "The tall, matte black cylindrical vacuum-insulated water bottle with a silver ring."
- Pragmatically Ambiguous: "I'm thirsty" (implying they want the water bottle).
Current state-of-the-art models like GroundingDINO often choke on these variations. They might get confused by too many adjectives or fail entirely to understand the intent behind a vague statement. This paper seeks to quantify this "prompt-conditioned robustness" and find ways to fix it.
Methods
The researchers built an evaluation pipeline that pairs OSOD models with a "Prompt Enhancement" module powered by a high-end VLM (GPT-5).
- Prompt Simulation: Instead of just using standard labels, they used a VLM to generate four distinct prompt styles (Underdetailed, Standard, Overdetailed, and Ambiguous) for objects in real-world images.
- Detection Models: They tested two leading OSOD architectures: GroundingDINO (which uses a transformer to fuse image and text) and YOLO-E (a faster, more lightweight model).
- Enhancement Strategies: They introduced two post-processing methods to "clean" the user's input before it hits the detector:
- Key Object Extraction: Distilling a long, rambling description into a concise noun phrase (e.g., "red sports car").
- Semantic Category Grounding: Mapping the user's intent to a standard category from a known taxonomy like COCO or LVIS.
Datasets
The study utilized DiverseAR and DiverseAR+, datasets consisting of 264 images captured from AR headsets and mobile devices in cluttered, real-world indoor environments (offices, bedrooms, etc.). Unlike standard datasets, these include "virtual overlays" and complex viewpoints typical of XR usage. The authors manually annotated these with bounding boxes to create a ground-truth reference for their experiments.
Key Insights and Ideas
- The "Over-Specification" Trap: Surprisingly, more detail isn't always better. For GroundingDINO, "overdetailed" prompts actually caused performance to drop. The model tries to satisfy every single adjective in the prompt; if the image is cluttered, it might get "distracted" by other objects that partially match those attributes.
- Detectors are Literal, VLMs are Contextual: OSOD models are great at matching words to pixels, but they are terrible at "reading between the lines." They cannot infer that "I'm thirsty" means "find the water bottle." However, VLMs excel at this reasoning, making them the perfect "middleman" for human-AI interaction.
- Performance Saturation: Once a prompt reaches a "minimum threshold of semantic clarity," adding more description provides diminishing returns. A "standard" prompt performs nearly as well as an "underdetailed" one, provided the core object name is present.
Key Results
The impact of VLM-based prompt enhancement was most dramatic in the "Pragmatically Ambiguous" category:
- Localization Accuracy: For GroundingDINO, mean Intersection over Union (mIoU) jumped from a dismal 35.84% (raw prompt) to 90.99% after Semantic Category Grounding—a 55% absolute gain.
- Confidence Scores: Model confidence increased by 41% on average when using enhanced prompts.
- Model Differences: YOLO-E proved more robust to "wordy" (overdetailed) prompts than GroundingDINO, likely due to its recall-oriented detection strategy, though GroundingDINO remained more precise overall when given clean input.
Limitations and Future Work
While the results are promising, the authors note several areas for growth:
- Scale: The study used a relatively small set of 264 images. Larger-scale benchmarking is needed to see if these trends hold across thousands of categories.
- Temporal Consistency: The experiments were done on static images. In real XR, the "prompt" and the "object" exist in a 3D video stream. Future work should look at how these strategies handle camera motion and changing light.
- Human-in-the-loop: The "user" prompts were simulated by a VLM. The next step is testing this with actual humans to see if the VLM-enhancement matches real-world intent and "cultural nuances" like idioms or metaphors.
Summary updated