Abstract
Conversational AI systems often struggle when users refer to entities using descriptive language rather than formal names. This paper introduces AltEntities, a large-scale dataset designed to benchmark the resolution of indirect referring expressions (REs). Using a novel "conversational cartoon" data collection method, researchers gathered over 42,000 REs across books, recipes, and music. Benchmarking reveals that while large language models (LLMs) like T5 achieve high accuracy when provided with background context, a significant gap remains compared to human performance, particularly when relying on internal model knowledge alone.
Introduction
In natural conversation, users frequently employ indirect referring expressions (REs) such as "the one with the little yellow balls on top" instead of "Simnel cake." These expressions arise from memory lapses, pronunciation difficulties, or a preference for descriptive attributes. Resolving these references requires mapping linguistic descriptions to external knowledge about entities. While traditional NLP has addressed direct or locational references, resolving REs based on shared "common ground" remains a significant challenge for modern AI.
The AltEntities Dataset
To address this, researchers at Google Research developed AltEntities, a dataset comprising 42,529 referring expressions linked to 6,247 entity pairs across three domains: Books, Recipes, and Music. To ensure the task is non-trivial, the dataset includes "hard negatives"—entity pairs with similar titles, overlapping Wikipedia descriptions, or shared metadata (e.g., the same author or genre).
Methodology: The Conversational Cartoon
A key contribution of this work is the data collection strategy. To elicit natural, casual language, the team employed a cartoon-bubble completion method.
Annotators acted as "Bob" in a three-panel comic, responding to "Alice's" clarification question. To prevent simple copy-pasting of text, a memory-recall setup was used: annotators studied entity information on one screen but wrote their responses on a separate screen without the source text visible. This encouraged the "fuzzy," attribute-based descriptions typical of human speech.
Modeling and Contextual Knowledge
The study evaluated several transformer-based architectures, including Universal Sentence Encoder (USE), Cross-Encoder BERT, BERT Joint, and T5 XL (3B parameters). A critical finding was the importance of external context.
When models were restricted to entity names only, accuracy plummeted to approximately 60%, indicating that internal LLM weights are insufficient for resolving specific or "long-tail" entities. Conversely, providing the models with the same background text seen by humans (the "Oracle" setting) boosted accuracy to 95%. In realistic scenarios using general Wikipedia data, performance ranged between 82% and 87%.
Key Results and Insights
- Top Performance: T5 XL emerged as the strongest model, particularly in the recipe domain, where it achieved 86.76% accuracy using unstructured background text.
- Joint Modeling: Models that performed a comparative analysis of both entity options simultaneously (e.g., BERT Joint) consistently outperformed those evaluating options in isolation.
- Cross-Domain Robustness: Linguistic patterns of indirect reference proved surprisingly universal; models trained on one domain (e.g., Books) maintained high performance when tested on others (e.g., Music).
- Error Analysis: Nearly half of the errors in the Books domain were attributed to "No Textual Overlap," where users referenced facts absent from the model's provided background text.
Conclusion and Future Directions
The AltEntities benchmark highlights the necessity of grounding conversational agents in external knowledge. Future work should explore scaling these systems to handle dozens of entity choices simultaneously and integrating Retrieval-Augmented Generation (RAG) to dynamically fetch the "common ground" required for resolution. Furthermore, the prevalence of visual descriptors (e.g., "the green one") suggests that multimodal models incorporating visual encoders will be essential for robust entity selection in real-world applications.
Summary updated