Reference and coreference in situated dialogue

Sharid Loáiciga, Simon Dobnik, David SchlangenNAACL 2021

TL;DR

This paper explores the challenges of annotating coreference in situated dialogue, where language is grounded in a shared visual context. The authors analyze two corpora, highlighting how visual grounding complicates traditional coreference resolution by introducing new types of references and requiring consideration of speakers' cognitive states. This work is crucial for developing more sophisticated vision-and-language systems that can truly understand and participate in situated conversations.

What Problem Does This Paper Address?

Traditional coreference resolution, a well-studied problem in Natural Language Processing (NLP), typically deals with textual data where entities are introduced and referred to within a linear, structured text. However, with the rise of vision-and-language (V&L) research, systems are increasingly expected to understand and generate language in the context of visual information. This paper tackles the challenge of extending coreference annotation to "situated dialogue," where conversations unfold with a shared visual scene. Unlike purely textual dialogues, situated dialogues involve references that can be grounded in the visual context, leading to complexities such as:

  • Visual grounding: Linguistic expressions can refer to objects or aspects of the visual scene, requiring a connection between language and perception.
  • Dynamic context: The visual context is not static; participants might have different perspectives or focus on different parts of the scene.
  • Speaker's cognitive state: Understanding references can depend on what each participant sees, believes, and intends, going beyond simple textual salience.

The authors argue that existing V&L corpora and annotation schemes are insufficient to capture these nuances, necessitating new approaches to coreference annotation that account for situated dialogue.

Methods

The paper presents a linguistic perspective on the challenges of coreference annotation in situated dialogue by analyzing pilot annotations of two corpora: Cups and Tell-me-more. The authors' approach involves:

  • Linguistic Analysis: Examining how referring expressions are used in situated dialogues and how they differ from purely textual contexts.
  • Annotation Scheme Adaptation: Starting from existing annotation schemes for textual coreference (e.g., Artstein and Poesio, 2006; Pradhan et al., 2007), they identify shortcomings when applied to situated dialogue.
  • Corpus Analysis: Investigating specific examples from the Cups and Tell-me-more corpora to illustrate the challenges. This includes analyzing:
  • Grounding and Referentiality: How references are tied to visual elements, including cases where object detectors might fail or where the degree of reference (e.g., a specific subset vs. all objects of a certain type) is ambiguous.
  • Speaker's Cognitive State: How participants' beliefs and intentions influence their references, especially when there's a mismatch in what they see (e.g., believing there's one missing cup when there are two).
  • Level of Specification: The granularity required for annotation, such as distinguishing between referring to individual objects versus groups or "stuff" (Caesar et al., 2018).
  • Information Status: How the visual context affects whether an entity is considered "discourse-new" or "discourse-old," deviating from purely textual patterns.

The paper does not introduce new computational models but focuses on the linguistic and annotation challenges that inform future system development.

Datasets

The paper analyzes data from two existing corpora:

  • Cups corpus: This corpus features conversations between two participants interacting over a visual scene with a table and cups. Participants have different views of the scene and must communicate to find unseen cups.
  • Tell-me-more corpus: This corpus consists of images accompanied by short, task-constrained dialogues (pseudo-dialogues) where participants describe images.

These corpora are used to illustrate the challenges of situated coreference annotation, rather than for training or evaluating new models.

Key Insights and Ideas

  • Visual context fundamentally alters coreference: The presence of a shared visual scene introduces new ways to refer to entities and requires grounding linguistic expressions in perceptual representations.
  • Speaker's mental state is critical: In situated dialogue, understanding references often depends on inferring what each participant knows, believes, and intends, especially when their visual perspectives differ. This goes beyond traditional discourse models.
  • Ambiguity in reference degree is common: Referring expressions can be underspecified, making it difficult to determine if a reference points to a specific object, a subset, or a general category, especially when grounded in a visual scene.
  • Standard coreference annotation is insufficient: Existing schemes, designed for text, struggle to capture the nuances of situated dialogue, particularly regarding visual grounding, speaker states, and information status.
  • Bridging and deixis are more prevalent: The visual context facilitates references that might be considered less common or more difficult in purely textual settings.

Key Results

The paper does not present quantitative results from model evaluations. Instead, its key contributions are qualitative:

  • Identification of specific annotation challenges: The authors detail several types of referring expressions and situations in the Cups and Tell-me-more corpora that are difficult to annotate using traditional methods. Examples include:
  • References to parts of objects not recognized by object detectors (e.g., "base of the tub").
  • Ambiguous references to subsets of objects (e.g., "your blue ones").
  • References to groups of objects not arranged in strict geometric lines (e.g., "rows of objects").
  • Situations where participants' differing beliefs about the visual scene lead to coreference errors.
  • Demonstration of the need for richer annotation schemes: The analysis highlights that a comprehensive annotation scheme for situated dialogue must account for visual grounding, speaker cognitive states, and the dynamic nature of the shared context.

Limitations and Future Work

  • Limitations:
    • The paper focuses on a linguistic perspective and pilot annotation, not on developing or evaluating computational models for coreference resolution in situated dialogue.
  • The analysis is based on two specific corpora, and the generalizability of the findings to other situated dialogue scenarios might need further investigation.
  • Future Work:
    • Developing richer annotation schemes that can capture the complexities of situated coreference.
  • Creating new corpora specifically designed for situated coreference annotation.
  • Building V&L systems that can effectively resolve coreferences in situated dialogue, leveraging visual grounding and speaker modeling.
  • Exploring how these challenges relate to broader V&L tasks like dialogue systems and social robotics.

Summary updated

BibTeX
@inproceedings{loaiciga2021reference,
  author = {Sharid Loáiciga and Simon Dobnik and David Schlangen},
  title = {{Reference and coreference in situated dialogue}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure