What Did You Refer to? Evaluating Co-References in Dialogue

Weinan Zhang, Yue Zhang, Hanlin Tang, Zhengyu Zhao, Caihai Zhu, Ting LiuACL 2021 · OpenReview

TL;DR

This paper introduces DEQA, a novel benchmark dataset and evaluation task for assessing how well dialogue models understand entity-level co-reference resolution within conversations. The authors demonstrate that even state-of-the-art dialogue models struggle with this task, highlighting a significant gap in their ability to truly comprehend conversational context beyond superficial pattern matching.

What Problem Does This Paper Address?

Modern conversational AI, like chatbots and virtual assistants, often relies on sequence-to-sequence models trained on vast amounts of dialogue data. While these models can generate seemingly coherent responses, it's difficult to ascertain if they genuinely understand the nuances of a conversation. Specifically, they struggle with linguistic structures like ellipsis, anaphora, and co-reference – how different mentions in a dialogue refer to the same entity. Current evaluation methods, which primarily focus on the quality of generated responses, don't directly measure this deeper understanding. This makes it hard to pinpoint model limitations and understand why a model might produce a good or bad response, as it could be due to genuine understanding or simply exploiting spurious patterns.

Methods

To directly measure co-reference understanding, the authors propose a question-answering (QA) approach. Instead of traditional co-reference resolution where models identify all mentions of an entity, they insert clarification requests (questions) into dialogues. The model's task is then to answer these questions, which implicitly tests its ability to resolve anaphora and ellipsis. This method is advantageous because:

  • It allows direct evaluation of existing dialogue systems without requiring separate co-reference resolution modules.
  • Responses to clarification questions are factual, facilitating automatic evaluation.
  • It's easier for crowd-sourcing compared to manual co-reference chain annotation.
  • The framework is extensible to other linguistic structures like event co-references and semantic relations.

The paper also reviews several representative neural end-to-end dialogue generation models, including HRED, vHRED, CVAE, Transformer, and DialoGPT, and analyzes their performance on the proposed task.

Datasets

The paper introduces DEQA (Dialogue Entity via Question Answering), a large-scale benchmark dataset. DEQA consists of:

  • An English dataset derived from DailyDialogue, containing 8,415 dialogues and 11,904 questions.
  • A Chinese dataset collected from Douban, comprising 6,203 dialogues and 10,387 questions.

Each dialogue in DEQA is annotated with one or more questions that probe for entity-level understanding, specifically focusing on ellipsis, anaphora, and co-reference phenomena. The answers can be entities, phrases, clauses, or fragments, and sometimes require summarizing information across multiple utterances.

Key Insights and Ideas

  • Co-reference is a Hard Problem for Dialogue Models: Even advanced models struggle significantly with understanding co-reference, indicating a fundamental limitation in their conversational comprehension.
  • QA as a Proxy for Understanding: Framing co-reference resolution as a question-answering task provides a direct and practical way to evaluate a model's grasp of dialogue context.
  • Pre-trained Models Show Promise: Models like DialoGPT, which leverage large-scale pre-training, generally outperform models trained from scratch, suggesting that pre-training is crucial for developing better dialogue understanding capabilities.
  • Attention Mechanisms Help: Models incorporating attention mechanisms (Static, Dynamic, ReCoSa) show improved performance, highlighting their utility in capturing relevant contextual information for co-reference.
  • Challenges Vary by Linguistic Structure: The paper suggests that clause and fragment types of co-references are particularly challenging for models.

Key Results

The paper reports that representative dialogue models achieve relatively low performance on the DEQA dataset, with EM (Exact Match) scores below 20% and F1 scores below 40% for both English and Chinese dialogues. For instance, DialoGPT, a strong pre-trained model, achieved the highest EM and F1 scores, but still demonstrated significant room for improvement. The results indicate that while models can generate coherent responses, their ability to accurately resolve co-references and answer clarification questions is limited.

Limitations and Future Work

The authors acknowledge that the current DEQA dataset primarily focuses on entity-level co-reference. Future work could involve annotating the dataset with questions related to more complex linguistic structures, such as event co-references and semantic relations. They also plan to release the DEQA dataset to facilitate further research in this area. The ethical considerations regarding data privacy and intellectual property rights for the Douban dataset are also discussed, with plans to obtain permission for redistribution.

Summary updated

BibTeX
@inproceedings{zhang2021what,
  author = {Weinan Zhang and Yue Zhang and Hanlin Tang and Zhengyu Zhao and Caihai Zhu and Ting Liu},
  title = {{What Did You Refer to? Evaluating Co-References in Dialogue}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=F5Vgwvco-ce},
}
Figure