Grounded Misunderstandings in Asymmetric Dialogue: A Perspectivist Annotation Scheme for MapTask

Nan Li, Albert Gatt, Massimo Poesio

TL;DR

This paper introduces a novel annotation scheme to precisely track how speakers and listeners in a collaborative dialogue understand (or misunderstand) references to objects, especially in situations where they have different information (asymmetric dialogue). They use this scheme with large language models (LLMs) to analyze a dataset of map-based dialogues, revealing that while outright misunderstandings are rare, subtle differences in how participants perceive object counts (multiplicity discrepancies) are a major source of hidden misalignments.

What Problem Does This Paper Address?

In everyday conversations, especially when working together on a task, people constantly try to ensure they're on the same page – this is called "grounding." Traditionally, researchers assumed that once a reference to an object (like "the red car") is made and acknowledged, both participants understand it to be the same object. However, this breaks down in "asymmetric settings" where participants have different information, like looking at slightly different maps. In these cases, they might think they agree, but actually be referring to different things. The HCRC MapTask corpus, a classic dataset for studying this, has annotations, but they don't fully capture these subtle, perspective-dependent misunderstandings. This paper aims to fill that gap by developing a way to meticulously track individual interpretations of references and analyze where these hidden misalignments occur.

Methods

The core of this work is a new "perspectivist annotation scheme." Instead of just noting if a reference is grounded, it captures:

  1. Speaker's Intended Landmark ID: What the speaker meant to refer to.
  2. Addressee's Interpreted Landmark ID: What the listener thought was being referred to.
  3. Five Binary Attributes: These describe the state of understanding for each reference, including whether it's a question about existence (is_quantificational), if the addressee noticed it (is_specified), if they understood it (is_accommodated), if they successfully linked it to a landmark (is_grounded), and if they imagined a landmark not on their map (is_imagined).

To apply this scheme at scale, they developed a "scheme-constrained LLM-in-the-loop annotation pipeline." This involves carefully crafting prompts for LLMs (specifically GPT-5) that guide them through the annotation process step-by-step, ensuring the output adheres to a strict JSON schema. They also introduced a "unified landmark ID" system to better handle discrepancies in landmark names and counts compared to the original MapTask annotations.

Datasets

The primary dataset used is the HCRC MapTask corpus. This corpus consists of dialogues where participants navigate a route using slightly different maps. The authors leveraged the existing annotations within this corpus (timed units, reference expressions, move tags) and enhanced them with their new perspectivist annotation scheme. They annotated approximately 13,000 reference expressions from the corpus.

Key Insights and Ideas

  • Hidden Misunderstandings: The most striking insight is that while outright "misunderstood" states are rare (especially after accounting for different names for the same object), a significant portion of apparent grounding masks subtle referential misalignment.
  • Multiplicity is Key: Discrepancies in the number of times a landmark appears on different maps (multiplicity discrepancies) are the biggest culprits for these hidden misunderstandings. When a landmark appears twice on one map but only once on another, participants often assume they're referring to the same thing, even if they're not.
  • "Running Repairs" are Crucial: The low rate of persistent misunderstandings suggests that participants are actively engaged in "running repairs" – constantly checking and clarifying to maintain alignment, rather than just making discrete error corrections.
  • LLMs as Annotators: The study demonstrates that LLMs, when guided by a well-defined, schema-constrained prompt, can achieve high accuracy in complex, perspective-dependent annotation tasks, opening doors for large-scale analysis.

Key Results

  • The LLM annotation pipeline achieved 94.4% RE-level accuracy on a human-verified subset of the data, with individual attribute accuracies ranging from 96.2% to 100%.
  • After unifying lexical variants (different names for the same object), the overall misunderstanding rate dropped significantly to 1.82%.
  • Multiplicity discrepancies accounted for a disproportionately high percentage of misunderstandings (48.5% of all misunderstandings), despite being a smaller portion of the total landmarks (7.3%).
  • Lexical discrepancies (different names for the same object) led to the longest reference chains, indicating that resolving naming mismatches requires more turns.

Limitations and Future Work

  • Single LLM: The annotation pipeline was tested with a single LLM (GPT-5). Future work should evaluate cross-model generalizability and inter-annotator agreement with multiple LLMs and human annotators.
  • Spatial Reasoning: The current scheme focuses on reference resolution but doesn't capture finer-grained spatial reasoning, path choices, or direction ambiguities.
  • Multimodality: The MapTask corpus was originally collected with potential for eye contact and prosody. This study relied solely on transcripts and maps, missing non-verbal cues that could influence grounding.
  • Future Directions: The authors plan to use their annotated dataset to establish benchmarks for evaluating (V)LLMs' ability to perform incremental grounding in various information settings, moving beyond simple object identification. They also suggest future evaluations could explicitly test LLMs' "omniscience" or perspective-taking abilities in dialogue.

Summary updated

Abstract

Collaborative dialogue relies on participants incrementally establishing common ground, yet in asymmetric settings they may believe they agree while referring to different entities. We introduce a perspectivist annotation scheme for the HCRC MapTask corpus (Anderson et al., 1991) that separately captures speaker and addressee grounded interpretations for each reference expression, enabling us to trace how understanding emerges, diverges, and repairs over time. Using a scheme-constrained LLM annotation pipeline, we obtain 13k annotated reference expressions with reliability estimates and analyze the resulting understanding states. The results show that full misunderstandings are rare once lexical variants are unified, but multiplicity discrepancies systematically induce divergences, revealing how apparent grounding can mask referential misalignment. Our framework provides both a resource and an analytic lens for studying grounded misunderstanding and for evaluating (V)LLMs' capacity to model perspective-dependent grounding in collaborative dialogue.
BibTeX
@inproceedings{li2026grounded,
  author = {Nan Li and Albert Gatt and Massimo Poesio},
  title = {{Grounded Misunderstandings in Asymmetric Dialogue: A Perspectivist Annotation Scheme for MapTask}},
  booktitle = {LREC},
  year = {2026},
  doi = {10.63317/59anbt78wyj7},
}
Figure