A method for the approximation of incremental understanding of explicit utterance meaning using predictive models in finite domains

David DeVault, David TraumNAACL 2013

TL;DR

This paper presents a novel method to approximate a dialogue system's "explicit understanding" of a user's partial utterance by leveraging an existing "predictive understanding" model. This approach allows systems to understand what has been said so far, even when full meaning is uncertain, offering a more robust and natural interaction experience.

What Problem Does This Paper Address?

Modern dialogue systems are increasingly aiming for "incremental understanding," meaning they try to process and understand user speech as it's being spoken, rather than waiting for the entire utterance. This allows for faster responses, more natural feedback (like backchannels), and better handling of interruptions.

There are two main ways to achieve incremental understanding:

  1. Predictive Understanding: At any point in a user's speech, the system tries to predict the full meaning of the complete utterance. This is useful for anticipating user needs and speeding up responses.
  2. Explicit Understanding: At any point, the system tries to determine the meaning that has been explicitly expressed so far in the partial utterance. This is crucial because sometimes predicting the full meaning is difficult or unreliable, but understanding what has already been said is still valuable for actions like highlighting mentioned items or providing grounding feedback.

The challenge is that while predictive models are well-established, directly achieving explicit understanding incrementally can be difficult and often requires extensive, costly annotation (e.g., aligning every word to its semantic meaning). This paper addresses the gap by proposing a way to approximate explicit understanding using the information already available in a predictive model, thereby reducing the need for expensive new annotations.

Methods

The core of the paper's contribution lies in deriving an "explicit subframe" from a predictive incremental NLU model called mxNLU. Here's how it works:

  • Predictive Model Foundation: The system starts with a predictive NLU model (mxNLU) that, given a partial ASR (Automatic Speech Recognition) result, predicts the most likely complete NLU frame for the entire utterance. This model assigns probabilities to different complete frames.
  • Frame Element Probabilities: The key insight is to use the probabilities assigned to complete frames to infer probabilities for individual frame elements (the semantic components within a frame). Specifically, for a given partial ASR result rj and a frame element e, the probability that e will be part of the correct complete frame Gu is calculated by summing the probabilities of all complete frames Si that contain e, conditioned on rj. This is represented by Equation (1).
  • Explicit Subframe Construction: To approximate explicit understanding, the authors propose selecting frame elements whose calculated probability (from Equation 1) exceeds a certain threshold (T). This creates an "explicit subframe" – a set of frame elements that are deemed highly likely to be present in the final meaning, based on the partial utterance so far. This is represented by Equation (2).
  • Thresholding for Precision/Recall Trade-off: The threshold T acts as a knob. A higher threshold leads to fewer, more confident frame elements (higher precision, lower recall), while a lower threshold includes more elements (lower precision, higher recall). This allows for tuning the approximation based on application needs.

Datasets

  • SASO-EN spoken dialogue system corpus: This corpus, collected from a system designed for practicing multi-party negotiation skills, was used for training and evaluation. It contains user utterances that have been audio-recorded, transcribed, and manually annotated with complete NLU output frames.
  • 3,826 training utterances
    • 449 testing utterances
  • Hand-annotated subframes: For evaluation purposes, the authors hand-annotated word-meaning alignments for 50 random utterances in the test set. These annotations represent human judgments of explicit incremental utterance meaning for specific utterance prefixes.

Key Insights and Ideas

  • Implicit to Explicit: The paper demonstrates that information about explicit meaning is implicitly encoded within a predictive understanding model. By carefully analyzing frame probabilities, we can "extract" this explicit meaning.
  • Cost-Effective Annotation: The proposed method significantly reduces the annotation burden. It only requires full utterance-to-frame annotations (which are already needed for the predictive model) rather than expensive word-to-meaning alignments.
  • Tunable Precision/Recall: The use of a probability threshold provides a flexible way to balance precision and recall for explicit understanding, allowing developers to tailor the system's behavior.
  • Complementary Strengths: This approach allows dialogue systems to combine both predictive and explicit incremental understanding, leveraging the strengths of each. Explicit understanding can be more reliable when prediction is uncertain.
  • "Subframes" as Explicit Meaning: The output of this method is not necessarily a complete NLU frame but a "subframe" – a subset of frame elements that accurately reflects what has been explicitly conveyed.

Key Results

  • Good Calibration: The derived frame element probabilities are shown to be "well-calibrated," meaning the probability assigned to a frame element closely matches its actual frequency of occurrence in the correct frame.
  • Strong Performance: When evaluated against hand-annotated subframes, the proposed method achieved an F-score of 0.75 at a threshold of 0.5-0.55. This is competitive with, and even slightly better than, the performance of the predictive NLU model itself on complete utterances (F-score of 0.56).
  • Higher Precision than Recall: As expected, increasing the probability threshold generally led to higher precision but lower recall in the explicit subframes, demonstrating the effectiveness of the threshold for tuning.
  • Explicit vs. Predictive: The explicit subframes were found to be better at recalling hand-annotated subframe elements than at recalling complete frame elements, confirming that the method is indeed capturing what has been said rather than just predicting the future.

Limitations and Future Work

  • Approximation, Not Perfect: The authors acknowledge that this is an approximation of explicit NLU. Because it's derived from a predictive model trained without detailed word-meaning alignments, it may occasionally include predictive aspects of meaning.
  • Need for Domain-Specific Models: The method relies on an existing predictive NLU model trained for a specific domain (SASO-EN). Its effectiveness would depend on the quality and coverage of that underlying model.
  • Future Work:
    • Explore other machine learning models that can leverage richer training data (e.g., word-meaning alignments).
  • Investigate further the combination of explicit and predictive techniques.
  • Compare the performance, annotation costs, and development costs of this approach with other methods, particularly those using word-aligned data and ensemble classifiers.

Summary updated

BibTeX
@inproceedings{devault2013method,
  author = {David DeVault and David Traum},
  title = {{A method for the approximation of incremental understanding of explicit utterance meaning using predictive models in finite domains}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure