Multimodal Response Obligation Detection with Unsupervised Online Domain Adaptation

Shota Horiguchi, Naoyuki Kanda, Kenji NagamatsuInterspeech 2019

TL;DR

This paper introduces a novel multimodal system for detecting when a dialogue robot should respond, combining visual, audio, and text cues. Crucially, it tackles the challenge of domain dependency by employing unsupervised online domain adaptation, allowing the robot to automatically adjust to new environments without explicit retraining.

What Problem Does This Paper Address?

Dialogue robots are designed to interact with humans, but a key challenge is determining when a human utterance is directed at the robot and requires a response. Utterances can be human-to-human conversations, monologues, or public announcements, and responding to these inappropriately can be disruptive. Traditional methods for "Response Obligation Detection" (ROD) often rely on explicit cues (like wake-up words), which limit usability, or use scenario-specific features that don't generalize. Furthermore, even implicit methods struggle when the robot operates in environments different from where it was trained (domain shift), especially due to the text modality's sensitivity to context. This paper aims to create a highly accurate and broadly applicable ROD system that can adapt to new environments on the fly.

Methods

The core of the proposed solution is a multimodal ROD detector that fuses information from three sources:

  • Visual-based detector: Analyzes facial features (position, size, orientation) of individuals in front of the robot using a 2-layer LSTM.
  • Audio-based detector: Extracts acoustic features (MFCC, PLP) and processes them with a 2-layer LSTM.
  • Text-based detector: Leverages Automatic Speech Recognition (ASR) to transcribe utterances (supporting both English and Japanese) and then uses word embeddings and a 1-layer LSTM.

The outputs from these detectors are combined using a simple average to produce a final response obligation probability.

The paper's main innovation lies in its unsupervised online domain adaptation (DA). This DA method addresses the domain dependency problem, particularly for the text modality, through two key mechanisms:

  1. Logistic Regression Update: The weights of the logistic regression layers for each modality are updated using unlabeled target domain data.
  2. Word Embedding Assignment: For words not seen in the source domain, a novel embedding assignment strategy is employed. This involves clustering existing word embeddings from the source domain and assigning new words to the closest cluster center based on whether the detector predicts a positive or negative response obligation. This helps the text modality adapt to new vocabulary and contexts.

Importantly, the LSTM parameters are kept fixed during DA to reduce computational cost.

Datasets

The authors collected their own datasets by operating a humanoid robot in two distinct real-world environments:

  • Station Dataset: Collected at a train station.
  • Building Dataset: Collected at a commercial building.

These datasets consist of synchronized audio and RGB video, with utterances annotated for response obligation. For evaluation, these datasets were divided into five subsets based on recording dates.

Key Insights and Ideas

  • Multimodality is Key: Combining visual, audio, and text information significantly boosts ROD accuracy compared to using individual modalities.
  • Text Modality's Value: Despite its domain dependency, the text modality is crucial for high accuracy. Avoiding it, as some prior work did, is detrimental to performance.
  • Domain Adaptation is Essential: The proposed unsupervised online DA effectively bridges the gap between source and target domains, recovering accuracy lost due to domain mismatch.
  • Novel Word Embedding Strategy: The method for assigning embeddings to unseen words is a critical component of the DA, allowing the text detector to handle new vocabulary in target domains.
  • Online Adaptation is Practical: The ability to adapt "on-the-fly" without needing labeled target data makes the system suitable for dynamic, real-world environments.

Key Results

  • In-domain performance: Using all three modalities (visual, audio, text) achieved the highest accuracy and ROC-AUC on both the Station and Building datasets, outperforming single or dual-modality systems. Text modality alone performed best among single modalities.
  • Out-of-domain performance: The unsupervised online DA significantly improved ROD performance when adapting from one domain to another. For example, in the Building to Station setting, accuracy was recovered by 0.074.
  • Embedding Assignment Effectiveness: The proposed word embedding assignment strategy for new words, especially with a sufficient number of clusters (K > 2), stably outperformed baseline methods (treating unknown words as a special symbol or using subword embeddings like SentencePiece).
  • Scalability: The detection performance improved as the number of target domain samples increased, demonstrating the method's ability to adapt automatically.

Limitations and Future Work

  • Computational Cost of DA: While LSTM parameters are fixed, the DA process itself involves clustering and updating embeddings and regression weights, which might still have computational implications for very rapid adaptation needs.
  • Dataset Specificity: The datasets were collected in specific "station" and "commercial building" environments. While the DA aims for generalization, performance in vastly different domains might require further investigation.
  • Future Work: The paper suggests that further exploration of embedding assignment strategies and more sophisticated DA techniques could yield even better results. The authors also implicitly highlight the need for robust ASR systems for diverse languages and accents.

Summary updated

BibTeX
@inproceedings{horiguchi2019multimodal,
  author = {Shota Horiguchi and Naoyuki Kanda and Kenji Nagamatsu},
  title = {{Multimodal Response Obligation Detection with Unsupervised Online Domain Adaptation}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure