TL;DR
This paper tackles the challenge of resolving referring expressions (like pronouns and deictics) in multi-modal dialogues, where speech is augmented by gestures and physical object manipulation. They introduce a novel approach that incorporates "haptic-ostensive" actions (physically interacting with an object) alongside pointing gestures, showing that these non-verbal cues are crucial for understanding references in human-robot or human-human interactions.
What Problem Does This Paper Address?
Understanding natural language is hard enough, but in real-world interactions, humans don't just speak – they point, they touch, they manipulate objects. This paper focuses on "co-reference resolution," the task of figuring out what a pronoun (like "it" or "he") or a deictic word (like "this" or "that") refers to. While most co-reference research has focused on written text, this paper addresses the much more complex domain of multi-modal dialogues. In these dialogues, especially those involving tasks like assisting the elderly, a significant portion of references are not made through speech alone but are introduced or clarified by physical actions. The authors highlight that previous computational models have largely ignored these non-verbal cues, particularly haptic actions, leaving a gap in our ability to build truly intelligent systems that can understand and participate in these rich interactions.
Methods
The core of this work lies in developing a co-reference resolution system that leverages multi-modal features. Given a pronoun or deictic, the system pairs it with potential antecedents found within a specific time window. These antecedents can be textual (though this paper focuses on non-textual ones) or, crucially, introduced by pointing gestures or "haptic-ostensive" (H-O) actions. H-O actions are defined as referring to an object by physically manipulating it (e.g., touching, holding, taking out).
The authors designed a novel annotation scheme to capture these gestures and actions within their multi-modal corpus. They then extract several features for co-reference classification:
- Time distance: How close in time the referring expression is to the gesture/action.
- Speaker agreement: Whether the speaker of the referring expression is the same as the performer of the gesture/action.
- Markable type agreement: Matching the type of the referring expression (e.g., object, place) with the type of the gesture's target.
- Number agreement: Ensuring the number of the referring expression matches the number of the target(s).
- Object agreement: For deictics, checking if a descriptive noun (e.g., "bowl" in "this big blue bowl") matches the target object.
- H-O Action type: Specific types of H-O actions (e.g., Touch, Hold, Take-Out) are used as features.
They train classification models (MaxEnt, Decision Tree, SVM) using these features to predict whether a gesture/action-antecedent pair is a true co-reference link.
Datasets
The research utilizes a subset of the ELDERLY-AT-HOME corpus. This corpus contains multi-modal data (video, voice, haptics, transcriptions) from human-human dialogues in an elderly care domain. The authors specifically focus on "Find tasks" within this corpus, which are time spans where participants collaborate to find objects. They also introduce their own multi-modal co-reference annotation scheme for this data, which includes marking pointing gestures, H-O actions, and their corresponding antecedents.
Key Insights and Ideas
- Haptic-ostensive actions are crucial: The paper's most significant contribution is demonstrating that physically interacting with objects (H-O actions) is a vital, yet previously underutilized, cue for co-reference resolution in multi-modal dialogues.
- Gestures dominate references: A striking finding is that approximately two-thirds of referring expressions in their corpus are introduced by pointing gestures or H-O actions, rather than speech alone. This underscores the importance of non-verbal communication.
- Short histories suffice: The best performing models for co-reference resolution in this context utilized very short temporal histories (around 2 seconds), suggesting that the most immediate gestures and actions are the most informative for resolving references.
- Task-specific annotation is key: The development of a detailed annotation scheme for gestures and H-O actions was essential for training and evaluating their models.
Key Results
The co-reference resolution system built using multi-modal features significantly outperforms a baseline model that only considers speaker agreement and immediate temporal proximity. The best models achieved an F-score of up to .747. While the percentage of correctly resolved pronouns/deictics (46%) might seem modest, the authors emphasize that this is a harder task than typical co-reference resolution because they are resolving references solely through gestures and H-O actions, not relying on textual antecedents. The performance of the three classification models (MaxEnt, Decision Tree, SVM) was comparable, with no single model showing a clear advantage.
Limitations and Future Work
The authors acknowledge that their current system does not handle textual antecedents. This means they are only resolving references that are introduced by gestures or H-O actions, not those that might have a spoken antecedent. Future work will involve integrating a module to handle textual antecedents, including type information from indefinite descriptions. They also plan to further investigate the role of timing and explore how to make the system fully multi-modal by combining both textual and non-textual reference resolution. Preliminary analysis suggests that gesture antecedents are much more immediate than textual ones, reinforcing the importance of short histories.
Summary updated