User Goal Change Model for Spoken Dialog State Tracking

Yi MaNAACL 2013

TL;DR

This paper proposes a novel Maximum Entropy Markov Model (MEMM) for spoken dialog state tracking that explicitly models user goal changes. By predicting when a user's goal might shift and incorporating this into the tracking process, the system can achieve more accurate and efficient inference of dialog states, leading to more natural and robust conversations.

What Problem Does This Paper Address?

Spoken dialog systems aim to understand and respond to user requests. A crucial component is "dialog state tracking," which infers the user's underlying goal and preferences throughout a conversation. Traditional approaches often assume a user's goal remains fixed, which is unrealistic. Users frequently change their minds, explore alternatives, or have unclear initial goals. This assumption leads to system failures when users deviate from expected paths, as illustrated by an example where the system fails to understand a user's request for flights on a different day. Existing methods struggle to handle these dynamic user goals, especially when dealing with noisy speech recognition (ASR) outputs and the inherent ambiguity in natural language. The paper aims to bridge this gap by developing a system that can adapt to evolving user goals.

Methods

The core contribution is a probabilistic framework based on a Maximum Entropy Markov Model (MEMM) that explicitly incorporates a user goal change variable. This model works in two steps:

  1. Predicting Goal Change: The system first predicts whether a user's goal has changed in a given dialog turn, using linguistic features and dialog context.
  2. Inferring Dialog State: The MEMM then utilizes this predicted goal change information to infer the most probable sequence of dialog states.

The proposed MEMM leverages N-best lists from ASR, considering both intra-dependencies within a single turn's hypotheses and inter-dependencies across turns. It uses designed feature functions that:

  • Prioritize common evidence when the goal is assumed to be unchanged.
  • Give more weight to current observations when a goal change is detected.
  • Incorporate confidence scores from ASR hypotheses.

The paper also briefly discusses approaches for detecting user goal change, suggesting methods like binary classification using linguistic features (e.g., specific phrases like "How about?") and dialog context (e.g., system returning no results).

Datasets

Not discussed in the paper. The paper refers to the "CMU Let's Go system" and a prototype "BuildByVoice" system for examples and evaluation context, but specific datasets for training or evaluation are not detailed.

Key Insights and Ideas

  • User goals are dynamic: The fundamental insight is that assuming a fixed user goal is a significant limitation in spoken dialog systems, leading to brittle behavior.
  • Explicitly modeling goal change is key: By predicting and incorporating user goal changes, systems can become more robust and handle complex negotiation strategies.
  • Leveraging ASR N-best lists: The proposed MEMM effectively exploits the rich, albeit noisy, information present in multiple ASR hypotheses, not just the top one.
  • Feature engineering for goal change: Identifying linguistic cues and dialog context can provide valuable signals for detecting user goal shifts.

Key Results

The paper proposes a probabilistic framework and a set of feature functions for dialog state tracking that explicitly models user goal changes. While quantitative results are not presented in detail, the authors claim that their approach leads to:

  • More efficient and accurate dialog state inference.
  • Better exploitation of mutual information in ASR N-best lists.
  • Improved handling of negotiation-style dialogs.
  • More natural and efficient conversations.

The paper uses illustrative examples (Figure 2, Figure 3, Figure 5) to demonstrate how the proposed model would handle situations where traditional methods would fail due to unexpected user goal shifts.

Limitations and Future Work

  • User Goal Change Detection: While the paper proposes the MEMM for state tracking with goal change, the subsection on "User Goal Change Detection" outlines potential approaches (MLP, SVM, Logistic Regression with linguistic features) but doesn't present a fully developed detection module or its evaluation.
  • Computational Complexity: The paper acknowledges that while partition-based methods offer scalability, the proposed MEMM's complexity for real-world domains is not fully detailed.
  • Domain Specificity: The examples are drawn from specific domains (flight booking, car configuration), and the generalizability of the proposed features and model to a wider range of domains is not explicitly discussed.
  • Future Work: The authors suggest that their approach opens up new possibilities for automated negotiation dialog policy learning. Further research could focus on developing and evaluating the user goal change detection module and exploring its application in more complex dialog management strategies.

Summary updated

BibTeX
@inproceedings{ma2013user,
  author = {Yi Ma},
  title = {{User Goal Change Model for Spoken Dialog State Tracking}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure