TL;DR
This paper is a tutorial proposal for the ACL 2020 conference, aiming to bridge the gap between classic grounding research in dialogue systems and the emerging complexities of multi-modal interactions. It proposes a structured approach to understanding how humans and increasingly sophisticated AI systems achieve "common ground" – shared understanding – across various communication channels, including vision and robotics.
What Problem Does This Paper Address?
The core problem this tutorial addresses is the challenge of achieving "common ground" in multi-modal dialogue systems. While traditional dialogue systems have focused on grounding in text-based conversations, modern systems increasingly integrate visual information, gestures, and even physical actions (in robotics). This shift necessitates a re-evaluation of grounding strategies, moving beyond simple confirmation actions to more nuanced ways of establishing mutual understanding in complex, multi-channel environments. The authors argue that existing research offers limited guidance for designing grounding modules in these cutting-edge systems, highlighting the need to revisit human-human communication insights in light of new technological capabilities.
Methods
This tutorial proposal outlines a structured exploration of grounding, rather than presenting a single novel method. The proposed approach involves three main topic areas:
- Grounding in human-human communication: Revisiting foundational work, particularly Brennan's characterization of grounding as seeking and providing evidence of mutual understanding, and Clark et al.'s two-phase model of communication (presentation and establishment of mutual beliefs).
- Grounding in dialogue systems: Reviewing established techniques for maintaining engagement, handling joint attention, turn-taking, and ambiguity resolution, often employing reinforcement learning for dialogue management.
- Grounding in multi-modal interactive systems: This is the core focus, exploring how signals like nodding, gestures, and gaze contribute to grounding. It will cover work on bridging vision and language through tasks like visual dialogue games, generating referring expressions for images, and visually grounded spoken language communication. The tutorial will also discuss how robots can demonstrate understanding through task execution and interactive learning.
Datasets
Not explicitly detailed as a primary focus of the tutorial proposal, but the authors reference several existing datasets and resources used in their cited works, including:
- Image-text discourse relations corpora: (e.g., Alikhani et al., 2019)
- Visual dialogue datasets: (e.g., Geman et al., 2015; Das et al., 2017)
- Crowdsourced resources and virtual reality systems: Used for collecting data for multimodal communication (e.g., Novikova et al., 2016).
The proposal emphasizes the need for new methods and data collection strategies tailored to multimodal grounding.
Key Insights and Ideas
- Grounding is more than just confirmation: The tutorial aims to move beyond simple "did you understand?" checks to exploring how systems can actively seek and offer evidence of understanding in a more dynamic and strategic way.
- Multi-modal signals are crucial: Gestures, gaze, and other non-verbal cues are not just supplementary but integral to establishing common ground, especially in human-robot interaction.
- Medium of communication matters: The choice of modality (text, vision, speech, action) significantly impacts how grounding is achieved and what techniques are effective.
- Human-human grounding offers valuable lessons: Insights from decades of research on how humans build shared understanding can inform the design of more sophisticated AI systems.
- The challenge of "lightweight practical action": Modern systems are increasingly capable of performing actions, which adds a new dimension to grounding that needs to be understood and leveraged.
Key Results
As this is a tutorial proposal, it does not present experimental results. Instead, it outlines the topics that will be covered, drawing upon existing research. The "results" are the insights and findings from the body of work the tutorial will synthesize.
Limitations and Future Work
The authors acknowledge that current cutting-edge systems, while increasingly multimodal, still face significant challenges in achieving robust common ground. They highlight the need for:
- More flexible and powerful grounding mechanisms: Moving beyond current limitations to handle complex content and diverse interaction scenarios.
- Catalyzing research and system building: The tutorial aims to inspire further work that directly explores creative and strategic ways for conversational agents to achieve mutual understanding.
- Addressing shortcomings of current vision and language models: The proposal points to existing datasets and tasks that reveal limitations in current models, suggesting areas for future development.
- Cross-domain and cross-interaction type solutions: Identifying common challenges and solutions that apply across different application domains and interaction types.
Summary updated