Correlation between model-based approximations of grounding-related cognition and user judgments

Klaus-Peter Engelbrecht, Sebastian MöllerInterspeech 2012

TL;DR

This paper introduces novel user models that capture cognitive aspects of spoken dialog systems (SDSs), specifically a user's belief about the system's state and their learning of system vocabulary. By correlating parameters derived from these models with user quality judgments, the authors demonstrate a more nuanced approach to evaluating SDS usability beyond simple behavioral metrics.

What Problem Does This Paper Address?

Evaluating complex spoken dialog systems (SDSs) is crucial but time-consuming and expensive, often relying on costly user studies. While user simulations offer a more efficient alternative, existing "user models" primarily focus on overt user behavior. This leaves a gap in understanding the internal cognitive processes that influence user experience and satisfaction. Specifically, the authors aim to bridge this gap by modeling two key cognitive aspects related to "grounding" in dialog: (1) the user's mental model of the system's current state and (2) how users learn to use the correct vocabulary and phrasing that the system understands. Understanding these cognitive elements could lead to more accurate and generalizable automatic evaluation methods.

Methods

The paper proposes two main models:

  • Belief Model: This model tracks the user's belief about the system's state, mirroring the system's internal state (e.g., filled slots for restaurant information like price range, food type, location). Unlike the system's state, the user's belief is updated not just by explicit user input but also by system feedback. The model is deterministic, simplifying implementation.
  • Learning Wordings Model: This model focuses on how users learn to use vocabulary understandable by the system. It distinguishes between speech understanding (SU) errors caused by incorrect vocabulary/expressions and those due to speech decoding. By analyzing these errors in conjunction with system feedback (e.g., indicated AVPs, successful AVPs, asked slots), the model aims to quantify a user's learning progress.

The authors then derive parameters from these models, such as the "gradient of distance" between the believed and desired system state over time, and a "perceived concept error rate." These parameters are then used in correlation analyses with user judgments.

Datasets

The study uses a dataset collected with the BoRIS restaurant information system. This dataset was gathered in a Wizard-of-Oz setting, where a human experimenter replaced the Automatic Speech Recognition (ASR) component, but ASR errors were simulated. The dataset includes transcripts, speech understanding results, task success annotations, and usability questionnaire ratings from 40 users performing various tasks.

Key Insights and Ideas

  • Cognition Matters: Modeling cognitive aspects like belief about system state and vocabulary learning provides valuable insights into user experience, going beyond simple behavioral metrics.
  • Belief State is Predictive: Parameters derived from the belief model (e.g., progress towards the goal) show significant correlations with user quality judgments, suggesting users are sensitive to how well they perceive the system understands their intent.
  • Perceived vs. Actual Errors: The "perceived concept error rate" (based on the user's belief state) correlates more strongly with user ratings than the actual concept error rate, highlighting the importance of the user's subjective experience of understanding.
  • Learning Vocabulary Impacts Acceptance: While the learning wordings model's results were less clear-cut, there's evidence that successful vocabulary learning, particularly in high-acceptance dialogs, positively influences user attitude.
  • Recency Effect: The recency of information (how recently an AVP was mentioned) influences its impact on user judgments, with more recent events having a stronger effect.

Key Results

  • Parameters derived from the belief model (e.g., GradDist_ALL, GradDist_END) showed correlations with user ratings (ACC, COE, EFF) comparable to standard predictors like #Turns and Task Success.
  • GradDist_END (progress in the last 3 turns) showed increased correlation with overall acceptance (ACC).
  • The perceived concept error rate (percCER) had stronger correlations with user ratings than the actual concept error rate (CER).
  • The parameter #Indicate (system indicating possible values) showed surprisingly strong negative correlations with user judgments, even stronger than #indiKnownAvps (indicating AVPs already known to the user). This suggests that while learning is important, frequent indications might be perceived negatively if not managed well.
  • Table 4 indicates that for dialogs with high acceptance ratings, the effect of knowing an AVP on error rates is much stronger than in poorly rated dialogs, suggesting that successful learning contributes to positive user attitudes.

Limitations and Future Work

  • Learning Wordings Model Complexity: The results for the learning wordings model were less clear than for the belief model. This is partly attributed to users often responding directly to system questions, leading to correct wordings that don't fully exercise learning capabilities. The paper notes that the system's NLU grammar was often designed based on these directed prompts.
  • Future Work: The authors suggest deriving more parameters, including subjective task success. They also plan to investigate how emotions impact the cognitive processes modeled. Additionally, the presented models could be integrated into user simulations for more realistic state updates and temporal modeling of speech understanding errors.

Summary updated

BibTeX
@inproceedings{engelbrecht2012correlation,
  author = {Klaus-Peter Engelbrecht and Sebastian Möller},
  title = {{Correlation between model-based approximations of grounding-related cognition and user judgments}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure