Seeing Isn't Resolving: Visual and Linguistic Disambiguation with Conversational Grounding

Edoardo Cecchinato, Shun Inadumi, Angel García Contreras, Giuseppe Riccardi, Koichiro YoshinoarXiv 2026 · arXiv

TL;DR

Researchers have introduced JAG-D, a new Japanese benchmark designed to test how well Visual Language Models (VLMs) resolve ambiguity using conversation history. The study reveals a striking asymmetry: while modern VLMs are excellent at using dialogue to fix linguistic gaps (like missing pronouns), they remain remarkably poor at resolving visual ambiguity, such as distinguishing between two identical-looking objects in a scene.

What Problem Does This Paper Address?

Visual grounding—the ability of an AI to locate an object in an image based on a text description—is a cornerstone of human-robot interaction. However, most existing benchmarks use "idealized" conditions where instructions are perfectly clear and unique.

In the real world, communication is messy. A human might say "Take that mug" in a room with three mugs, or simply say "Pass it" (an example of zero anaphora, common in Japanese where subjects/objects are often omitted). To succeed, a robot must look back at the conversation history to understand which "mug" or "it" the human is referring to. This paper addresses the lack of benchmarks that jointly control for both visual and linguistic ambiguity within a conversational context.

Methods

The authors developed a two-pronged evaluation framework:

  1. Ambiguity Recognition: Can the model detect if an instruction is ambiguous given only the current image and text?
  2. Ambiguity Resolution: Can the model correctly ground the object once it is provided with the full dialogue history?

They tested several state-of-the-art models, including specialized grounding models like GLIP and MDETR, and general-purpose VLMs like Qwen3-VL and Gemma3.

To dig deeper into why models improve with dialogue, they performed a Dialogue Intervention Analysis. They fed the models "counterfactual" histories—such as shuffled turns, reversed orders, or completely irrelevant text—to see if the models were actually reasoning about the conversation or just picking up on surface-level patterns.

Datasets

The study introduces the JAG-D (Japanese evaluation for Ambiguous visual Grounding with Dialogue) dataset.

  • Size: 480 carefully curated instances.
  • Source: Images are from the J-CRe3 egocentric human-robot interaction dataset.
  • Categories: Ambiguities are split into Visual (Same Object, Same Attribute) and Linguistic (Pronoun Ambiguity, Zero Anaphora).
  • Dialogue Modes: Instances are labeled as Resolving (history clarifies the target), Non-resolving (history is irrelevant), or Supportive (history adds redundant but helpful context).
Example of visual and linguistic ambiguity resolution through dialogue history.

Key Insights and Ideas

  • The Visual-Linguistic Asymmetry: Models are surprisingly adept at recognizing linguistic underspecification (e.g., realizing a sentence is missing an object) but fail to notice when multiple regions in an image satisfy a description. They "see" the scene but don't "reason" about the visual competition between objects.
  • Zero Anaphora is a Solvable Problem: In Japanese, omitting the object is a major hurdle for AI. JAG-D shows that providing dialogue context leads to massive performance jumps (up to +60% accuracy) for these cases, suggesting models are quite good at "filling in the blanks" from text.
  • Content Over Form: The intervention analysis proved that models rely on the semantic content of the dialogue. When the history was replaced with neutral or irrelevant text, performance tanked, whereas shuffling the turn order had a much smaller negative impact.
Figure 2: Change in grounding performance under tar- geted interventions on the dialogue history for Qwen. Points show the mean per-sample ∆IoU vs. the full- dialogue baseline; horizontal bars are 95% confidence intervals across samples. Ambiguous instances (blue) are substantially more sensitive to interventions than non-ambiguous instances (orange). Extended results appear in Appendix D.

Key Results

  • Recognition Gap: Multilingual VLMs (Qwen3-VL, Gemma3) achieved near-ceiling recall (93-100%) on linguistic ambiguity but collapsed to 32-54% on visual ambiguity categories.
  • Grounding Boost: Adding dialogue context nearly doubled the accuracy for models like GLIP (26.9% → 53.3%) and MDETR (25.4% → 54.0%).
  • Top Performer: Qwen3-VL was the strongest overall, reaching 79.0% [email protected] when enhanced with dialogue, compared to 63.5% with the instruction alone.
  • Persistent Bottleneck: Even with dialogue, visual ambiguity remains a "blind spot." Accuracy for the "Same Object" category remained significantly lower than for linguistic categories across all models.

Limitations and Future Work

The authors acknowledge that the JAG-D benchmark is relatively small (480 instances) due to the high cost of expert manual verification. Furthermore, the study focuses exclusively on Japanese; while the findings regarding visual vs. linguistic reasoning likely generalize, the specific impact of zero anaphora is language-dependent.

Future work should aim to develop models that treat perception and discourse as jointly informative sources, rather than using dialogue as a simple post-hoc filter for visual predictions. The goal is to move toward agents that actively seek clarification when they "see" ambiguity, rather than just guessing.

Summary updated

Abstract

Visual grounding benchmarks usually do not consider the contextual uncertainty of real-world scenarios. In this study, we construct a novel benchmark, the Japanese evaluation for Ambiguous visual Grounding with Dialogue (JAG-D) dataset. It consists of human-robot conversational multimodal interactions covering different types of ambiguity, arising from visual and linguistic categories. We evaluate the capabilities of different state-of-the-art Visual Language Models (VLMs) for ambiguity recognition and resolution. We compare dialogue context grounding performance for visual and linguistic ambiguity resolution. Our experiments show that VLMs can substantially improve the resolution of linguistic ambiguity, especially Japanese zero anaphora, by using dialogue context. We further conduct a comparative analysis to understand whether grounding improvements arise from the semantic content of the dialogue or from its conversational structure.
BibTeX
@misc{cecchinato2026seeing,
  author = {Edoardo Cecchinato and Shun Inadumi and Angel García Contreras and Giuseppe Riccardi and Koichiro Yoshino},
  title = {{Seeing Isn't Resolving: Visual and Linguistic Disambiguation with Conversational Grounding}},
  year = {2026},
  eprint = {2611.21183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21183},
}
Figure