ADVISER: A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents

Chia-Yu Li, Daniel Ortega, Dirk Väth, Florian Lux, Lindsey Vanderlyn, Maximilian Schmidt, Michael Neumann, Moritz Völkel, Pavel Denisov, Sabrina Jenne, Zorica Kacarevic, Ngoc Thang VuACL 2020

TL;DR

This paper introduces ADVISER, an open-source, Python-based toolkit for building multi-modal (speech, text, vision) conversational agents that are also socially engaged. It addresses the scarcity of flexible, extensible platforms for developing such complex agents, aiming to foster collaborative research across technical and non-technical domains.

What Problem Does This Paper Address?

Developing sophisticated conversational agents, especially those that go beyond simple task completion to engage users on a social and emotional level, is a significant challenge. While many commercial systems (like Alexa, Siri) and some academic prototypes exist, there's a lack of accessible, open-source toolkits that support multi-modal input (speech, text, vision) and incorporate social signals like emotion and engagement. Existing toolkits often focus on core dialog components, are not open-source, or are built on less accessible platforms like .NET, hindering broader adoption and collaborative research, particularly for linguists and cognitive scientists. ADVISER aims to fill this gap by providing a unified, flexible, and user-friendly platform.

Methods

ADVISER builds upon a previous text-based toolkit, extending it with multi-modal capabilities and social signal processing. Key methodological aspects include:

  • Modularity and Service-Oriented Architecture: The system is decomposed into independent, interchangeable services (e.g., NLU, ASR, NLG, emotion recognition) that communicate via a publish/subscribe mechanism. This allows for flexible pipeline construction and distributed deployment.
  • Multi-modal Integration: ADVISER synchronizes signals from speech, text, and vision inputs using the publisher/subscriber pattern, addressing challenges in handling varying latencies.
  • Social Signal Processing:
    • Emotion Recognition: Leverages text, audio, and vision modalities to predict user emotions (e.g., happy, angry, sad) and arousal/valence levels.
  • Engagement Level Prediction: Uses gaze tracking (via webcam and OpenFace) to infer user disengagement.
  • Backchanneling: An acoustic module predicts backchannel cues (e.g., "uh-huh") from speech to signal listener acknowledgment.
  • Speech Processing: Integrates end-to-end ASR (using ESPnet) and a FastSpeech-based TTS system for voice output, optimized for speed and quality.
  • Dialog Management: Includes handcrafted and reinforcement learning (RL) policies for task-oriented dialogs, along with a rule-based affective policy for emotional responses. A user simulator is provided for training and evaluation.
  • External Information Access: Supports querying SQL databases, APIs, and knowledge bases like Wikidata.
  • Affective NLG: Generates system responses that adapt to the user's detected emotional state, aiming for more empathetic interactions.

Datasets

The paper mentions the following datasets:

  • IEMOCAP: Used for training emotion recognition models.
  • LibriSpeech, Switchboard, TED-LIUM 3, AMI, WSJ, Common Voice 3, SWC, VoxForge, M-AILABS: Used for training the end-to-end English ASR model.
  • LJSpeech: Used for training the FastSpeech and WaveGAN TTS models.

The paper does not introduce new datasets but leverages existing ones for its modules.

Key Insights and Ideas

  • Unified Platform for Multi-modal and Socially-Engaged Agents: The core insight is the creation of a single, open-source toolkit that seamlessly integrates multi-modal inputs with social signal processing, a combination often fragmented in existing research.
  • Service-Oriented Architecture for Flexibility: The publish/subscribe model is crucial for breaking away from rigid pipelines, enabling easy integration of new services, domain-specific modules, and distributed deployment.
  • Social Signals as Key to Human-like Interaction: The paper emphasizes that incorporating emotional states and engagement levels is vital for developing agents that are not just functional but also friendly and likeable, moving beyond purely task-oriented systems.
  • Bridging Technical and Non-Technical Researchers: By making the toolkit Python-based, modular, and user-friendly, ADVISER aims to empower a broader community, including linguists and cognitive scientists, to contribute to conversational AI research.

Key Results

The paper focuses on presenting the toolkit and its architecture rather than specific quantitative results from end-to-end evaluations. However, it highlights:

  • Comprehensive Functionality: ADVISER offers a wide range of modules for speech processing, NLU, dialog state tracking, dialog policies, NLG, and social signal processing.
  • Flexibility and Extensibility: The modular design allows users to easily combine services, adapt them, and extend the system for new domains and functionalities.
  • Support for Distributed Systems: The architecture is designed to facilitate distribution across multiple machines, enabling efficient handling of resource-intensive tasks.
  • Example Use Cases: Demonstrates the toolkit's capability with example domains like lecturers, courses, weather, and dining hall information, showcasing multi-domain and multi-modal interactions.

Limitations and Future Work

  • Limitations:
    • The current implementation of turn-taking is a naive end-of-utterance detection.
  • The affective NLG service uses templates, which might limit the expressiveness of emotional responses.
  • The paper focuses on task-oriented applications, not open-ended social chatbots like XiaoIce.
  • Future Work:
    • Incorporating more sophisticated turn-taking models.
  • Expanding emotion recognition to include visual features.
  • Integrating multiple training datasets for emotion recognition.
  • Developing more advanced affective policies and NLG capabilities.
  • Exploring more complex dialog management strategies.

Summary updated

BibTeX
@inproceedings{li2020adviser,
  author = {Chia-Yu Li and Daniel Ortega and Dirk Väth and Florian Lux and Lindsey Vanderlyn and Maximilian Schmidt and Michael Neumann and Moritz Völkel and Pavel Denisov and Sabrina Jenne and Zorica Kacarevic and Ngoc Thang Vu},
  title = {{ADVISER: A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure