Towards Natural Language Understanding of Partial Speech Recognition Results in Dialogue Systems

Kenji Sagae, Gwen Christian, David DeVault, David TraumNAACL 2009

TL;DR

This paper explores enabling dialogue systems to understand user utterances before they are fully spoken. By training specialized Natural Language Understanding (NLU) models on partial Automatic Speech Recognition (ASR) outputs, they demonstrate that systems can achieve significant understanding accuracy early in an utterance, paving the way for more natural and responsive human-computer conversations.

What Problem Does This Paper Address?

Traditional spoken dialogue systems operate on a rigid turn-taking model: they wait for the user to finish speaking, then process the complete utterance through ASR and NLU before responding. This can feel unnatural and inefficient, especially in mixed-initiative dialogues where humans naturally engage in sub-utterance level turn-taking and feedback. The core problem this paper addresses is how to equip dialogue systems with incremental processing capabilities, allowing them to begin understanding and potentially acting on user input as it's being spoken, rather than waiting for completion. This is crucial for creating more fluid and human-like conversational agents.

Methods

The authors investigate incremental NLU by training and evaluating specialized NLU models. Their approach involves:

  • Partial ASR Output: Leveraging the capability of ASR systems to provide partial results as the user speaks.
  • Specialized NLU Models: Instead of using a single NLU model trained on complete utterances, they train separate models for different lengths of partial ASR input (e.g., models for 1-word, 2-word, up to 10-word partial utterances). This is referred to as "length-specific NLU models."
  • Feature Engineering: The NLU models use features derived from the ASR output, including individual words (bag-of-words), word bigrams, pairs of words, and the total number of words.
  • Contextual Features (Model 3): They also explore incorporating features from the preceding system utterance to capture dialogue context, though this proved less impactful.
  • Maximum Entropy Models: The underlying NLU models are based on maximum entropy, a standard machine learning technique for classification tasks.

Datasets

The research utilizes a corpus of 4,500 user utterances from a negotiation scenario involving the location of a medical clinic.

  • Development Set: Used for training the NLU module.
  • Test Set: Used for final evaluation.
  • Data Split: Sessions were randomly assigned to development and test sets to ensure all utterances within a session remained together. Out-of-domain utterances were assigned a "garbage" frame.

Key Insights and Ideas

  • Early Understanding is Feasible: The most significant insight is that substantial NLU accuracy can be achieved with partial ASR results, even before the user finishes speaking. This validates the premise of incremental processing for NLU.
  • Specialized Models Outperform General Models: Training NLU models specifically for partial utterances of certain lengths yields better results than using a single model trained on complete utterances. This suggests that the linguistic cues present in early parts of an utterance are distinct and can be effectively learned by dedicated models.
  • Diminishing Returns with Length: The benefits of incremental understanding are most pronounced for shorter partial utterances (e.g., 4-5 words). As utterances get longer, the time saved by early processing decreases, and the accuracy gains become more modest.
  • Contextual Features Less Impactful (in this study): While incorporating dialogue context features showed some promise, they did not significantly boost NLU accuracy in this particular experiment, suggesting that alternative methods for context integration might be needed.

Key Results

  • Baseline NLU (complete ASR): Achieved an F-score of 0.76 for frame elements when trained and tested on complete ASR utterances.
  • Incremental NLU (Model 2 - length-specific): Achieved an F-score of 0.71 for partial utterances of 5 words, compared to 0.66 for the baseline model. For 4-word partials, the F-score was 0.67, significantly higher than the baseline's 0.58.
  • Stability: Figure 3 shows that frames produced by specialized models for partial utterances of 8 words or more closely match the frames for complete utterances, indicating high stability. However, frames for 6-word partials, while accurate (Figure 2), are still often revised when the full utterance is available.

Limitations and Future Work

  • Context Integration: The authors acknowledge that the context features used in Model 3 did not provide substantial benefits and suggest that other ways of representing context or dialogue state might be more effective. This is an area they are actively investigating.
  • System Integration: The paper focuses on the NLU component. Future work will involve extending the NLU model to utilize different context features and exploring how dialogue agents can leverage these early interpretations for more dynamic interactions, such as interrupting or finishing user sentences.
  • Stability vs. Accuracy Trade-off: While partial utterances of 6 words show good accuracy, Figure 3 indicates they are still often revised. This suggests a potential trade-off between early interpretation and final accuracy that needs careful management in a live system.

Summary updated

BibTeX
@inproceedings{sagae2009towards,
  author = {Kenji Sagae and Gwen Christian and David DeVault and David Traum},
  title = {{Towards Natural Language Understanding of Partial Speech Recognition Results in Dialogue Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure