Introduction
Common grounding—the process by which interlocutors reach a mutual understanding—is a cornerstone of human communication. While humans navigate this effortlessly, AI agents struggle in environments that mirror real-world complexity. This paper introduces the OneCommon dataset, a benchmark designed to test grounding under two realistic constraints: continuous attributes and partial observability.
The Challenge of Realistic Grounding
Existing dialogue research often simplifies the grounding problem by providing agents with full observability (identical views) or categorical contexts (discrete labels like "red" or "square"). In contrast, real-world scenarios involve significant ambiguity. Is a dot "dark gray" or "black"? Does my partner see the same cluster I do?
The authors argue that these simplifications make the task "too easy" for models, failing to capture the strategic depth of human coordination required when information is incomplete and attributes are nuanced.
Methods: The Collaborative Referring Task
To study this, the researchers developed a task where two agents must identify a shared object within a 2D grid of seven dots. These dots vary continuously in size and shade. Crucially, each agent has a slightly different perspective, meaning some dots are visible to both while others are unique to one individual.
Data Collection
The researchers collected 6,760 dialogues using Amazon Mechanical Turk. Participants were required to use natural language to find a common dot and select it simultaneously. A turn-taking mechanic was implemented to ensure clean dialogue data, and bonuses were provided for successful coordination.
Key Insights from Human Coordination
Analysis of the OneCommon corpus revealed several sophisticated strategies used by humans that current AI models struggle to replicate:
- Spatial Pragmatics: Humans often describe "triangles" or clusters of dots to ground their partner, even when no literal geometric shapes exist. They use the relative shape of the cluster to establish a shared frame of reference.
- Nuanced Language: The continuous nature of the attributes led to frequent use of hedges and modifiers like "slightly," "roughly," and "maybe."
- Joint Saliency: There is a strong bias toward discussing "extreme" objects (e.g., the darkest or largest dot) first, as these are most likely to be noticed by both parties.
- Negative Grounding: Under partial observability, learning what a partner cannot see is as vital as identifying what they can. A "no" response is a critical tool for pruning the search space.
Results and Baseline Performance
The authors evaluated several neural architectures on a target selection task: predicting which dot a human picked based on the dialogue and view. While a Relation Network—which specifically models spatial relationships—achieved 43.1% accuracy, it remained far behind human performance.
- Random Guessing: 14.3%
- Context-Only (MLP): 27.9% (reflecting the power of saliency bias)
- Context + Dialogue (Relation Net): 43.1%
- Human Performance: 82.5% – 90.8%
Conclusion
The OneCommon dataset exposes a massive gap in current AI's ability to coordinate in uncertain, continuous environments. The 40%+ performance gap between the best neural models and humans highlights that current dialogue systems lack the ability to truly ground language in a shared, partially-observable context. This work provides a fundamental testbed for developing next-generation agents capable of nuanced, strategic communication.
Summary updated