Abstract
Establishing common ground is fundamental to human communication, yet it remains a significant challenge for artificial intelligence. This paper presents a large-scale annotated corpus designed to track common grounding through reference resolution in a shared visual task. By providing over 40,000 annotated referring expressions across 5,191 dialogues, this work establishes a benchmark for evaluating whether dialogue systems possess a genuine understanding of entities or merely exhibit surface-level fluency.
Introduction
In collaborative dialogue, participants must continuously synchronize their internal states to reach a mutual understanding, a process known as common grounding. Failure in this process—such as misidentifying a referred object—leads to communication breakdown. While modern neural dialogue models demonstrate high linguistic fluency, they often lack robust grounding, particularly in partially-observable contexts where agents have different perspectives. This research addresses this gap by decomposing common grounding into the measurable subtask of reference resolution: identifying which specific entities a speaker intends to describe at each step of a conversation.
Methodology: Reference Resolution Annotation
The study utilizes a collaborative game where two players must identify a mutually visible dot within a 2D grid, despite having different viewpoints. To analyze how players reach agreement, the researchers developed a two-step annotation pipeline.
First, Markable Detection identifies independent referring expressions within the dialogue (e.g., "the large black dot"). Second, Referent Identification involves human annotators mapping these expressions to specific dots on the screen. To account for inherent linguistic ambiguity, each expression received at least three independent human judgments. This approach captures "reasonable disagreements," allowing the dataset to reflect the uncertainty present in natural language.
Dataset Characteristics
The resulting corpus extends the OneCommon dataset, comprising 40,172 referring expressions and 103,894 individual human judgments. Analysis of the data reveals that terms for color and size are highly pragmatic and relative. For instance, a dot's classification as "black" depends on the surrounding context rather than absolute RGB values. This highlights the necessity for models to perform contextual reasoning rather than simple pattern matching.
Key Insights and Experimental Results
The researchers evaluated baseline GRU-based neural networks on three tasks: target selection (TSEL), reference resolution (REF), and dialogue generation (DIAL).
- Joint Training Benefits: Models trained simultaneously on reference resolution and the final task performed better (69.09% accuracy) than those trained on the end-task alone (67.79%). This suggests that understanding what is being discussed is a prerequisite for successful collaboration.
- The Complexity of Grouping: While models excelled at identifying single dots or entire sets, they struggled with intermediate groups (2–6 dots). These cases require human-like pragmatic reasoning and spatial grouping.
- Right for the Wrong Reasons: A critical finding was that baseline models often reached the correct final target despite failing to resolve references correctly throughout the dialogue. This underscores the importance of granular evaluation through reference resolution to ensure true understanding.
Conclusion and Future Work
This annotated corpus provides a vital tool for peering into the "black box" of dialogue systems. By quantifying reference resolution, researchers can pinpoint exactly where a model's understanding of common ground fails. Future work will focus on improving model handling of complex pragmatic expressions and developing systems capable of active clarification when grounding becomes ambiguous.
Summary updated