The MonAMI reminder: a spoken dialogue system for face-to-face interaction

Jonas Beskow, Jens Edlund, Björn Granström, Joakim Gustafson, Gabriel Skantze, Helena TobiassonInterspeech 2009

TL;DR

This paper introduces the MonAMI Reminder, a multimodal spoken dialogue system designed to assist elderly and disabled individuals with daily organization. It tackles the challenge of making advanced technology accessible by integrating familiar tools like paper calendars with embodied conversational agents (ECAs) and innovative interaction techniques like head pose tracking.

What Problem Does This Paper Address?

Information and communication technologies (ICT) are increasingly integrated into our lives, but many elderly and disabled individuals risk being excluded due to complex interfaces and a lack of user-centered design. This paper addresses the need for ICT services that are not only accessible but also cater to the specific preferences and requirements of these user groups. The MonAMI project aims to develop and test new services, with this paper focusing on face-to-face interaction within the MonAMI context. The core challenge is to relieve users of cognitive, visual, and motor demands, shifting the interaction metaphor from desktop manipulation to natural, embodied conversation.

Methods

The MonAMI Reminder employs a user-centered design approach, informed by in-depth interviews with potential users. Key methodological innovations include:

  • Multimodal Interaction: The system integrates a familiar paper calendar with a digital pen for input, an embodied conversational agent (ECA) for spoken interaction, and web access for broader accessibility. This "augmented paper calendar" allows users to maintain their preferred organizational methods while gaining the benefits of digital reminders.
  • Dynamic Grammars for ASR: To overcome the limitations of static speech recognition models with potentially out-of-vocabulary terms (like specific event names or locations), the system uses dynamic grammars. These grammars are updated in real-time based on the content of the user's calendar, allowing for more robust recognition of user queries related to their schedule.
  • Face-to-Face Interaction and Attention Control: The system explores the use of an ECA for notifications and interaction. It investigates how head pose tracking can be used to model user attention and control the dialogue flow. This includes features like only listening when the user looks at the ECA, providing attentional feedback, and pausing or calling for attention based on gaze.
  • Higgins Platform: The system is built on the Higgins platform, a flexible, distributed architecture that allows components to run asynchronously and communicate via XML-encoded messages. This facilitates the integration of readily available, off-the-shelf components alongside custom-developed research modules.

Datasets

Not discussed in the paper. The paper focuses on the design and architecture of the system, with user interviews informing the design process.

Key Insights and Ideas

  • Bridging the Digital Divide with Familiarity: The most significant insight is the power of integrating new technology with existing, familiar tools. By allowing users to continue using paper calendars, the system significantly lowers the barrier to adoption for those who find purely digital interfaces daunting.
  • Dynamic Grammars are Crucial for Real-World Spoken Dialogue: The paper highlights how dynamic grammars, updated from the user's calendar, are essential for handling the unpredictable nature of user queries in a personalized domain like scheduling. This moves beyond static, pre-defined vocabularies.
  • Embodied Agents Enhance Accessibility and Trust: The use of an ECA is not just for novelty; it's presented as a way to make interactions more intuitive and potentially build trust, especially for users who may be hesitant about technology.
  • Head Pose Tracking for Natural Turn-Taking: The exploration of head pose tracking for attention control demonstrates a sophisticated approach to making human-computer interaction more natural and less demanding on the user. It allows the system to be more context-aware and less intrusive.

Key Results

The paper presents a system architecture and discusses the design principles and solutions under investigation. While specific quantitative results from user trials are not detailed in this excerpt, the paper indicates that user trials are ongoing. The authors report positive reception of similar ECA prototypes in previous projects, suggesting a promising direction for user engagement. The described multimodal approach and dynamic grammar system are presented as solutions to identified user needs and technical challenges.

Limitations and Future Work

  • User Trial Data: The paper acknowledges that user trials are currently underway, implying that the full impact and effectiveness of the MonAMI Reminder are yet to be definitively quantified.
  • Web Interface Synchronization: A noted limitation is that changes made via the web interface to the Google Calendar will not be reflected in the paper calendar, requiring careful management to avoid discrepancies.
  • Future Work: The authors are investigating the use of "look-to-talk" for user attention control in ongoing user trials, comparing it with traditional push-to-talk methods. Further research will likely focus on refining the ECA's interactional abilities and assessing its impact on user experience and task success. The multilingual capabilities of the system are also a key consideration for future development.

Summary updated

BibTeX
@inproceedings{beskow2009monami,
  author = {Jonas Beskow and Jens Edlund and Björn Granström and Joakim Gustafson and Gabriel Skantze and Helena Tobiasson},
  title = {{The MonAMI reminder: a spoken dialogue system for face-to-face interaction}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure