TL;DR
This paper investigates whether expert ratings of spoken dialogue systems (SDSs) can effectively predict user satisfaction, aiming to reduce the cost and effort of collecting user feedback. The authors found that while expert ratings are highly correlated with user ratings, current methods for predicting user satisfaction from expert ratings do not yet achieve acceptable performance. However, they conclude that expert ratings are a viable and more practical alternative to user ratings for many applications.
What Problem Does This Paper Address?
Intelligent User Interfaces (IUIs) that leverage spoken dialogue systems (SDSs) are becoming increasingly common. To make these systems more user-adaptive, it's crucial to understand their quality from the user's perspective. Traditionally, this quality is assessed through user satisfaction (US) ratings. However, collecting these ratings is expensive and time-consuming, as it requires users to actively provide feedback during or after interactions, which can disrupt the dialogue flow and deter participation.
An alternative is to use expert ratings, where trained individuals evaluate recorded dialogues. These expert ratings are less costly to obtain but might not perfectly reflect the actual user experience. This paper addresses the gap between these two rating methods by exploring whether expert ratings can be used to reliably predict user satisfaction, thereby offering a more efficient way to assess SDS quality.
Methods
The authors explore two main approaches to predict user satisfaction (US) using expert ratings (Interaction Quality, IQ) and other interaction parameters:
- Model Exchange: This approach uses statistical classifiers trained on IQ to predict US. The core idea is that since IQ and US are highly correlated, a model trained on IQ might generalize to predict US.
- Belief-Based Sequential Recognition: This method employs a Hidden Markov Model (HMM)-like structure where IQ is treated as a latent variable. It models the sequential nature of dialogues, updating a "belief state" of user satisfaction after each exchange, taking into account the previous state and the observed IQ. This approach also considers action-dependency, where system actions influence the belief update.
For classification, they utilize a range of standard algorithms: Support Vector Machines (SVMs) with cubic and RBF kernels, Naive Bayes, Naive Bayes with kernel, and Rule Induction.
Datasets
The study utilizes a corpus collected by Schmitt et al. (2011b) from a lab user study involving 38 users interacting with the "Let's Go Bus Information" system. This corpus contains:
- 128 calls, totaling 2,897 system-user exchanges.
- Both User Satisfaction (US) ratings and Interaction Quality (IQ) ratings, each on a 1-5 scale.
- Automatically derived interaction parameters from dialogue modules (e.g., ASR confidence, turn counts, system questions) at three levels: exchange, dialogue, and window.
This corpus is notable for containing both user and expert ratings for the same interactions, making it ideal for studying the relationship between them.
Key Insights and Ideas
- Strong Correlation: Expert ratings (IQ) and user ratings (US) are significantly correlated, as evidenced by moderate Cohen's Kappa values and strong Spearman's Rho values. This suggests that experts can indeed capture a substantial portion of the user experience.
- Expert Ratings are More Reliable: The paper consistently shows that IQ recognition (predicting expert ratings) is more reliable and accurate than US recognition (predicting user ratings), as indicated by higher UAR, Kappa, and Rho values for IQ.
- Prediction Limitations: Despite the strong correlation, current methods for predicting user satisfaction from expert ratings (Model Exchange and Belief-Based Sequential Recognition) do not achieve "acceptable" performance in terms of Unweighted Average Recall (UAR).
- Expert Ratings as a Practical Alternative: Given the higher reliability and lower cost of obtaining expert ratings, the authors conclude that they are a suitable replacement for user ratings in many practical scenarios, especially when automatic interaction improvement is the goal.
Key Results
- Reference Experiment (US Recognition): The baseline for direct US recognition (using models trained on US) achieved a maximum UAR of 0.50 with Rule Induction.
- Model Exchange: This approach, trained on IQ and evaluated on US, achieved a maximum UAR of 0.34 across various classifiers, performing worse than the reference experiment.
- Belief-Based Sequential Recognition (Action-Independent): This method achieved a maximum UAR of 0.33, also underperforming the reference experiment.
- Belief-Based Sequential Recognition (Action-Dependent): This variant showed marginal improvements over the action-independent version, with a maximum UAR of 0.35, but still did not surpass the reference experiment.
- IQ Recognition Performance: The best IQ recognition performance (using Rule Induction) achieved a UAR of 0.55, a Kappa of 0.68, and a Rho of 0.79, demonstrating the higher reliability of expert ratings.
Limitations and Future Work
- Acceptable Prediction Threshold: The primary limitation is that the proposed prediction methods (Model Exchange and Belief-Based Sequential Recognition) did not reach an "acceptable" UAR for predicting user satisfaction. The authors suggest that while expert ratings are similar to user ratings, the current models are not yet precise enough for direct prediction in all cases.
- Unbalanced Data: The unbalanced nature of the dataset (unequal distribution of ratings across classes) is identified as a potential reason for the observed results, affecting metrics like UAR.
- Future Work: The authors encourage further research into other parameters for dialogue quality assessment, such as emotion recognition. They also implicitly suggest exploring more advanced prediction models or feature sets to bridge the gap between expert and user ratings more effectively. The paper highlights that expert ratings are a more practical and reliable choice for many applications, even if direct prediction of user satisfaction remains a challenge.
Summary updated