Incrementally Tracking Reference in Human/Human Dialogue Using Linguistic and Extra-Linguistic Information

Casey Kennington, Ryu Iida, Takenobu Tokunaga, David SchlangenNAACL 2015

TL;DR

This paper presents an incremental model for resolving referring expressions in human-human dialogue that integrates linguistic cues with extra-linguistic information like gaze. The model achieves state-of-the-art results on a more natural, less constrained Japanese dialogue corpus, demonstrating its potential for real-time interactive systems.

What Problem Does This Paper Address?

Human communication heavily relies on referring expressions (REs) to identify objects and entities in the shared environment. Accurately resolving these references is crucial for natural human-computer interaction. Existing systems often struggle with complex, natural dialogue because they either:

  • Are tested in highly constrained, artificial settings (like Wizard-of-Oz experiments).
  • Do not process references incrementally, meaning they wait until an entire utterance is spoken before attempting resolution.
  • Fail to effectively integrate multi-modal information such as gaze or pointing gestures.
  • Are too slow for real-time interaction.

This paper aims to bridge this gap by applying and evaluating an incremental reference resolution model on a more natural, human-human dialogue corpus, incorporating linguistic and extra-linguistic cues.

Methods

The core of this paper is the application and evaluation of the Simple Incremental Update Model (SIUM). Key aspects of the SIUM include:

  • Incremental Processing: The model processes dialogue word-by-word, updating its interpretation of the referent at each step. This allows for earlier disambiguation and a more natural interaction flow.
  • Multi-modal Integration: SIUM can incorporate various information sources, including:
  • Linguistic Features (Ling): Standard language-based features.
  • Task-Specific Features (TaskSp): Information relevant to the specific task, such as whether an object was recently moved or has a mouse cursor over it.
  • Gaze Information (Gaze): Features derived from where participants are looking, such as the object gazed at longest or most recently.
  • Property-Based Grounding: The model grounds linguistic expressions to the world by linking words to object properties (e.g., shape, size, location). This is a key departure from previous methods that relied on direct concept matching.
  • Context Model for Initial Prior: A novel addition is the use of a context model to establish an initial prior probability distribution over potential referents before the RE even begins, leveraging saliency information.

Datasets

The paper uses the REX corpus, a collection of human-human dialogue data from collaborative Tangram puzzle-solving tasks. This corpus is notable for:

  • Being a more natural, less constrained setting than previously used datasets.
  • Involving Japanese language, which is head-final and differs typologically from German (used in prior work).
  • Containing a significant percentage of pronoun utterances (around 32%).
  • Providing annotations for utterances, object focus, pointing gestures, and segmentation of referring expressions.

The authors compare their results to previous work on the same corpus (Iida et al., 2011) and also mention using a separate part of the REX corpus (N2009-11) for development.

Key Insights and Ideas

  • Incremental processing is crucial for natural dialogue: Waiting for the entire utterance to be spoken is unrealistic for interactive systems. SIUM's word-by-word approach is a significant step towards more fluid interactions.
  • Property-based grounding is a powerful mechanism: Linking language to the world via object properties, rather than direct concept matching, allows SIUM to handle a wider range of references (including exophoric and deictic) within a single framework.
  • Extra-linguistic cues significantly improve accuracy: Integrating gaze and task-specific features, even in a simplified property-based manner, leads to substantial performance gains.
  • Saliency matters even before the utterance: The initial context model, which predicts potential referents based on saliency before the RE begins, provides valuable prior information.
  • Robustness across languages and modalities: The model's ability to perform well on Japanese data and integrate different modalities without reformulation highlights its flexibility.

Key Results

  • State-of-the-art performance: The SIUM model achieves better referential accuracy than previous approaches on the REX corpus, outperforming both combined and separated models from Iida et al. (2011).
  • Incremental metrics show promise: The model demonstrates respectable incremental behavior, picking the correct referent early in the RE and rarely changing its mind once a decision is made.
  • Efficiency: The model is designed to work in real-time, a key advantage for interactive systems.
  • Feature efficiency: SIUM achieves strong results using fewer features than some previous models, particularly in the gaze domain.

Limitations and Future Work

  • Pronoun resolution challenges: While SIUM handles pronouns, the paper notes that they were more difficult to resolve in this work compared to previous studies, possibly due to the Japanese pronoun "sore" having ambiguous anaphoric and demonstrative uses and less contextual information being leveraged compared to prior work.
  • Property computation: Although SIUM uses fewer features, these properties still need to be computed. Future work could investigate models that ground language directly from raw world information (e.g., vision) without explicit property extraction.
  • Collaborative reference: The current evaluation focuses on isolated REs. Future work should explore how SIUM handles collaborative reference resolution that unfolds over multiple turns.
  • Group objects: The current evaluation primarily focuses on references to single objects; resolving references to groups of objects is left for future work.

Summary updated

BibTeX
@inproceedings{kennington2015incrementally,
  author = {Casey Kennington and Ryu Iida and Takenobu Tokunaga and David Schlangen},
  title = {{Incrementally Tracking Reference in Human/Human Dialogue Using Linguistic and Extra-Linguistic Information}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure