Practical Application of Domain Dependent Confidence Measurement for Spoken Language Understanding Systems

Mahnoosh Mehrabani, David Thomson, Benjamin SternNAACL 2018 · DOI

Abstract

Spoken Language Understanding (SLU) systems in commercial environments face significant challenges from background noise and linguistic ambiguity. This paper proposes a practical framework for building domain-dependent confidence models that integrate traditional Automatic Speech Recognition (ASR) features with high-level semantic and domain-specific knowledge. Experimental results on real-world customer service data demonstrate significant improvements in error rejection, achieving relative performance gains of up to 72% in specific dialogue states.

Introduction

In commercial SLU systems, such as virtual assistants and automated phone menus, "False Accepts" (FA)—where the system incorrectly interprets user input—lead to poor customer experiences and broken workflows. To mitigate this, systems employ a confidence measure to decide whether to accept an interpretation or reject it for clarification or human intervention. While most research focuses on general ASR features, this study investigates how domain-specific semantic features can enhance model robustness and accuracy without being overly aggressive in rejecting valid inputs.

System Architecture

The proposed system integrates a confidence model at the end of the SLU pipeline. This model processes inputs from both the ASR and Natural Language (NL) components to generate a final confidence score, which is then compared against a threshold to determine if the utterance should be accepted or rejected.

SLU Confidence Model Architecture

The architecture relies on three primary feature categories:

  1. ASR Features: Derived from Word Confusion Networks (WCN), including "delta" scores (comparing the best path to "garbage" models) and frame counts.
  2. Semantic/NL Features: Including intent classification scores and bag-of-words distributions.
  3. Domain-Specific Logic: Tailored features such as checking for digit consistency in number-heavy inputs or specific keywords like "oh" in digit strings.

Methodology

The authors developed specific models for three distinct dialogue states: Opening (open-ended help requests), Number Capture (digit strings), and Confirmation (yes/no intents). Maximum Entropy (MaxEnt) models were selected as the primary classifiers due to their efficiency and strong performance with limited labeled datasets. While Neural Networks and Random Forests were tested, they did not provide a significant enough edge to justify their higher computational costs in a production environment.

Results and Discussion

The performance was evaluated using FA-Rej plots, measuring the False Accept rate against the Rejection rate. The integration of NL features provided substantial improvements across all domains, particularly when the system was tuned to a 20% rejection rate.

Performance Improvement in the Confirmation Domain

At a 20% rejection rate, the relative performance improvements over ASR-only baselines were:

  • Opening Domain: 29.98% improvement.
  • Number Capture Domain: 27.92% improvement.
  • Confirmation Domain: 72.46% improvement.

The dramatic gain in the Confirmation domain (shown above) highlights the power of NL classification scores in simple tasks where ASR confidence alone may be unreliable. The study also found that context is critical; features effective for the "Opening" state, such as bag-of-words, were less relevant for "Number Capture," where sequence length and specific ambiguous words were more predictive.

Conclusion

This research demonstrates that domain-dependent confidence modeling significantly enhances SLU reliability by leveraging semantic information. While complex models like Neural Networks were tested, MaxEnt remains the practical choice for production environments due to lower latency and training requirements. Future work will focus on scaling these models as more application-specific data becomes available to potentially leverage more complex non-linear features.

Summary updated

BibTeX
@inproceedings{mehrabani2018practical,
  author = {Mahnoosh Mehrabani and David Thomson and Benjamin Stern},
  title = {{Practical Application of Domain Dependent Confidence Measurement for Spoken Language Understanding Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-3023},
}
Figure