Parallel processing of interruptions and feedback in companions affective dialogue system

Jaakko Hakulinen, Markku Turunen, Raúl Santos de la Camara, Nigel CrookInterspeech 2010

TL;DR

This paper introduces the "How-Was-Your-Day" (HWYD) prototype, a dialogue system designed for more natural, affective interactions. It tackles the challenge of handling user interruptions and providing timely feedback by employing parallel processing paths and a dedicated Interruption Manager, moving beyond traditional pipeline architectures.

What Problem Does This Paper Address?

Traditional spoken dialogue systems often rely on rigid, sequential "pipeline" architectures. This approach, while effective for task-oriented dialogues with clear turns, struggles with more natural, fluid conversations where users might interrupt, provide emotional cues, or expect immediate feedback. The authors highlight the need for systems that can handle these complexities to foster more engaging, long-term relationships with users, moving beyond simple task completion. Specifically, they aim to create a system that can react flexibly to user activity, provide constant feedback, and gracefully manage interruptions, all while maintaining a natural conversational flow.

Methods

The HWYD prototype is built upon a message-passing framework called Inamode, connecting 15 distinct modules. The core innovation lies in its architecture, which deviates from a strict pipeline by implementing parallel processing paths and a specialized Interruption Manager (IM).

  • Parallel Processing Paths: Instead of a single sequential flow, the system utilizes multiple processing paths that operate concurrently. This allows for different aspects of user input (e.g., speech recognition, emotion analysis) to be processed simultaneously.
  • Main Dialogue Loop: This path handles the standard, more comprehensive processing of user input, involving modules like ASR (Automatic Speech Recognition), DAT (Dialogue Act Tagger), DM (Dialogue Manager), and NLG (Natural Language Generator).
  • Feedback and Interruption Loop: This parallel loop is designed for immediate reactions. It leverages partially processed information to enable timely feedback and handle interruptions. Key modules here include ATT (Acoustic Turn-Taking), AEC (Acoustic Emotion Classifier), and the IM.
  • Interruption Manager (IM): This central component orchestrates the system's response to interruptions. It monitors user activity (via ATT) and can trigger immediate feedback from the ECA (Embodied Conversational Agent) or instruct other components (like DM and ASM) to re-plan their actions when a user interrupts a system tirade. The IM also manages system state transitions.
  • Affective Components: The system incorporates modules for emotional analysis (AEC, SA - Sentiment Analyser, EM - Emotional Model) and affective strategy generation (ASM) to enable more nuanced and emotionally aware interactions.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Decoupling Processing Loops: The separation of a "main dialogue loop" from a "feedback and interruption loop" provides a clear conceptual and technical model for handling both comprehensive dialogue processing and immediate reactive behaviors.
  • The Interruption Manager as a Key Orchestrator: The IM is crucial for bridging the gap between traditional pipeline processing and the need for real-time responsiveness. It acts as a supervisor, ensuring timely feedback and graceful handling of user interruptions without disrupting the core dialogue flow.
  • Parallelism for Naturalness: Implementing parallel processing paths allows for a more flexible and timely response to user input, particularly for affective cues and interruptions, which are vital for creating a natural conversational experience.
  • Component Invisibility: A significant design achievement is that individual components largely remain unaware of the parallel processing loops. They process input according to their specifications, and the same output can be utilized by both loops, simplifying component design.

Key Results

The paper focuses on the architectural design and the demonstration of the HWYD prototype's capabilities rather than presenting quantitative performance metrics. The key "results" are qualitative:

  • The HWYD prototype successfully demonstrates the ability to react to speech in a timely manner.
  • It provides constant feedback via an animated character (ECA).
  • It effectively handles user interruptions, allowing for more natural dialogue flow.
  • The system can elicit more information from users and attempt to positively influence their emotional state.

Limitations and Future Work

  • Update Frequency Categorization: The authors note that the Interruption Manager doesn't neatly fit into the update frequency categories proposed by Shlangen and Skantze [8], suggesting a need for further refinement of such models.
  • Component Re-planning: While the IM can instruct other components to re-plan actions, the paper implies that the complexity of this re-planning process could be a challenge.
  • Future Work: The paper suggests that the presented architecture provides a solid foundation for further research into more sophisticated affective dialogue systems. The authors also point to the need for more detailed analysis of the system's performance and user experience.

Summary updated

BibTeX
@inproceedings{hakulinen2010parallel,
  author = {Jaakko Hakulinen and Markku Turunen and Raúl Santos de la Camara and Nigel Crook},
  title = {{Parallel processing of interruptions and feedback in companions affective dialogue system}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure