Online SLU model adaptation with a partial oracle

Pierre Gotab, Geraldine Damnati, Frederic Bechet, Lionel Delphin-PoulatInterspeech 2010

TL;DR

This paper introduces a novel method for continuously updating Spoken Language Understanding (SLU) models in deployed dialogue systems without requiring manual transcription or annotation of new data. By leveraging user feedback as a "partial oracle," the system can adapt its models online, significantly reducing the cost and effort associated with maintaining and improving these systems.

What Problem Does This Paper Address?

Deployed Spoken Dialogue Systems (SDS) need to adapt to evolving user behavior and new services. Traditionally, this adaptation involves collecting new user interactions, manually transcribing and annotating them, and then retraining the Automatic Speech Recognition (ASR) and SLU models. This process is costly and time-consuming, leading to infrequent updates and systems that quickly become outdated. The paper addresses the challenge of efficiently and cost-effectively updating SLU models in real-time, bridging the gap between the need for dynamic adaptation and the prohibitive cost of traditional methods.

Methods

The core innovation is the "partial oracle" approach. Instead of a human expert providing definitive transcriptions and semantic labels (a "full oracle"), the system uses the end-users of the deployed SDS as a "partial oracle." This partial oracle provides binary feedback: a "yes" or "no" indicating whether the system's predicted semantic label for a user's utterance was correct.

The proposed algorithm works as follows:

  1. Bootstrap: Initial SLU models (a set of binary classifiers, one for each possible semantic label) are trained on a pre-existing, fully annotated corpus.
  2. Online Adaptation: As new user utterances are collected from the deployed system, they are automatically transcribed using the existing ASR model.
  3. Partial Oracle Feedback: The SLU model predicts a semantic label for the utterance. The system then queries the partial oracle (simulated in this paper) to determine if the prediction was correct.
  4. Model Update:
    • If the partial oracle confirms the prediction ("yes"), the utterance (along with its automatic transcription and the correct label) is added to the training data for the corresponding classifier.
  5. If the partial oracle indicates an incorrect prediction ("no"), the utterance (with its automatic transcription) is added as a negative example to the training data of the predicted label and as a positive example to the training data of the true (but unknown to the system) label. This "wrong" feedback is crucial for learning.
  6. Retraining: The SLU classifiers are periodically retrained using the augmented training data.

The paper also explores adapting the ASR language model using these automatically transcribed utterances, further enhancing the system's adaptability.

Datasets

The experiments were conducted on data collected from a real deployed call-routing SDS by France Telecom.

  • Bootstrap Corpus: Used for initial training of ASR and SLU models.
  • Corpus 1 & Corpus 2: Used as pools of candidate utterances for adaptation. These were collected over different time periods (October 2008 - April 2009 and May 2009).
  • Test Corpus: A separate set of utterances (May 2009) used for evaluation.

The paper notes that 17% of the test utterances fall into a "garbage" class due to issues like false voice activity detection, out-of-domain speech, or uncovered utterances. The Word Error Rate (WER) on the test corpus was 47.7%.

Key Insights and Ideas

  • User as a "Partial Oracle": The most significant insight is the effective use of user feedback as a low-cost supervision signal. This bypasses the need for expensive manual annotation.
  • Value of "No" Feedback: The "YES+NO" partial oracle strategy significantly outperforms the "YES only" strategy. This highlights that learning from incorrect predictions (negative examples) is as crucial, if not more so, than learning from correct ones for robust model adaptation.
  • Online Adaptation is Feasible: The paper demonstrates that online adaptation of SLU models is achievable and effective, even with automatically generated transcriptions.
  • ASR Adaptation Benefits SLU: Adapting the ASR language model, even with potentially noisy automatic transcriptions, can lead to further improvements in SLU performance.

Key Results

  • SLU Adaptation: The "Partial Oracle (YES+NO)" strategy achieved nearly half of the performance gain possible with a "Full Oracle" (manual annotation) while requiring no additional manual annotation. The Interpretation Error Rate (IER) was significantly reduced compared to the "Partial Oracle (only YES)" approach.
  • ASR Adaptation: While the Word Error Rate (WER) of the ASR model itself showed only minor improvements with adaptation, the SLU performance (measured by IER) saw a significant decrease when ASR adaptation was incorporated, especially with the "YES+NO" partial oracle strategy.

Limitations and Future Work

  • Simulation of Partial Oracle: The experiments simulated the partial oracle by using manually annotated true labels. In a real deployment, obtaining this feedback might require explicit user interaction or more sophisticated log analysis.
  • ASR Transcription Quality: The performance of the adaptation heavily relies on the quality of the automatic transcriptions from the ASR system. A high Word Error Rate (47.7% in this study) can introduce noise that might hinder adaptation.
  • Future Work: The authors suggest exploring more sophisticated methods for obtaining partial oracle feedback and investigating how to better leverage the "wrong" feedback to improve both ASR and SLU models simultaneously. They also mention the potential for applying this approach to other components of SDS.

Summary updated

BibTeX
@inproceedings{gotab2010online,
  author = {Pierre Gotab and Geraldine Damnati and Frederic Bechet and Lionel Delphin-Poulat},
  title = {{Online SLU model adaptation with a partial oracle}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure