TL;DR
Researchers have introduced JAG-D, a new Japanese benchmark designed to test how well Visual Language Models (VLMs) resolve ambiguity using conversation history. The study reveals a striking asymmetry: while modern VLMs are excellent at using dialogue to fix linguistic gaps (like missing pronouns), they remain remarkably poor at resolving visual ambiguity, such as distinguishing between two identical-looking objects in a scene.
What Problem Does This Paper Address?
Visual grounding—the ability of an AI to locate an object in an image based on a text description—is a cornerstone of human-robot interaction. However, most existing benchmarks use "idealized" conditions where instructions are perfectly clear and unique.
In the real world, communication is messy. A human might say "Take that mug" in a room with three mugs, or simply say "Pass it" (an example of zero anaphora, common in Japanese where subjects/objects are often omitted). To succeed, a robot must look back at the conversation history to understand which "mug" or "it" the human is referring to. This paper addresses the lack of benchmarks that jointly control for both visual and linguistic ambiguity within a conversational context.
Methods
The authors developed a two-pronged evaluation framework:
- Ambiguity Recognition: Can the model detect if an instruction is ambiguous given only the current image and text?
- Ambiguity Resolution: Can the model correctly ground the object once it is provided with the full dialogue history?
They tested several state-of-the-art models, including specialized grounding models like GLIP and MDETR, and general-purpose VLMs like Qwen3-VL and Gemma3.
To dig deeper into why models improve with dialogue, they performed a Dialogue Intervention Analysis. They fed the models "counterfactual" histories—such as shuffled turns, reversed orders, or completely irrelevant text—to see if the models were actually reasoning about the conversation or just picking up on surface-level patterns.
Datasets
The study introduces the JAG-D (Japanese evaluation for Ambiguous visual Grounding with Dialogue) dataset.
- Size: 480 carefully curated instances.
- Source: Images are from the J-CRe3 egocentric human-robot interaction dataset.
- Categories: Ambiguities are split into Visual (Same Object, Same Attribute) and Linguistic (Pronoun Ambiguity, Zero Anaphora).
- Dialogue Modes: Instances are labeled as Resolving (history clarifies the target), Non-resolving (history is irrelevant), or Supportive (history adds redundant but helpful context).
Key Insights and Ideas
- The Visual-Linguistic Asymmetry: Models are surprisingly adept at recognizing linguistic underspecification (e.g., realizing a sentence is missing an object) but fail to notice when multiple regions in an image satisfy a description. They "see" the scene but don't "reason" about the visual competition between objects.
- Zero Anaphora is a Solvable Problem: In Japanese, omitting the object is a major hurdle for AI. JAG-D shows that providing dialogue context leads to massive performance jumps (up to +60% accuracy) for these cases, suggesting models are quite good at "filling in the blanks" from text.
- Content Over Form: The intervention analysis proved that models rely on the semantic content of the dialogue. When the history was replaced with neutral or irrelevant text, performance tanked, whereas shuffling the turn order had a much smaller negative impact.
Key Results
- Recognition Gap: Multilingual VLMs (Qwen3-VL, Gemma3) achieved near-ceiling recall (93-100%) on linguistic ambiguity but collapsed to 32-54% on visual ambiguity categories.
- Grounding Boost: Adding dialogue context nearly doubled the accuracy for models like GLIP (26.9% → 53.3%) and MDETR (25.4% → 54.0%).
- Top Performer: Qwen3-VL was the strongest overall, reaching 79.0% [email protected] when enhanced with dialogue, compared to 63.5% with the instruction alone.
- Persistent Bottleneck: Even with dialogue, visual ambiguity remains a "blind spot." Accuracy for the "Same Object" category remained significantly lower than for linguistic categories across all models.
Limitations and Future Work
The authors acknowledge that the JAG-D benchmark is relatively small (480 instances) due to the high cost of expert manual verification. Furthermore, the study focuses exclusively on Japanese; while the findings regarding visual vs. linguistic reasoning likely generalize, the specific impact of zero anaphora is language-dependent.
Future work should aim to develop models that treat perception and discourse as jointly informative sources, rather than using dialogue as a simple post-hoc filter for visual predictions. The goal is to move toward agents that actively seek clarification when they "see" ambiguity, rather than just guessing.
Summary updated