TL;DR
This paper explores how to predict user satisfaction and student learning in a spoken dialogue tutoring system using the PARADISE framework. They found that traditional system-generic parameters, while useful in other dialogue systems, were less effective for predicting satisfaction in this tutoring context. However, these parameters, along with tutoring-specific ones and even user affect, proved valuable for predicting student learning.
What Problem Does This Paper Address?
The increasing development of spoken dialogue tutoring systems aims to bridge the gap between human and computer tutors. A key challenge in evaluating these systems is accurately measuring their performance. Traditionally, user satisfaction has been a primary metric, often assessed through surveys. However, collecting this data can be expensive and time-consuming. The PARADISE framework offers a solution by using interaction parameters to predict performance. This paper investigates the applicability of PARADISE to a specific spoken dialogue tutoring system, ITSPOKE, and explores whether different types of parameters (system-generic, tutoring-specific, and user affect) are more effective for predicting either user satisfaction or student learning, especially in a context where learning, not just satisfaction, is the primary goal.
Methods
The authors employ the PARADISE framework, which uses multivariate linear regression to predict a target performance variable (user satisfaction or student learning) from a set of interaction parameters. They define three categories of parameters:
- System-Generic Parameters: These are standard metrics used in many dialogue systems, such as speech recognition quality (word error rate), dialogue efficiency (total turns, elapsed time), and task completion.
- Tutoring-Specific Parameters: These are tailored to the tutoring context and include measures like student correctness (e.g., "Correct," "Incorrect," "Partially Correct" turns) and the number of essay revisions.
- User Affect Parameters: These capture the user's emotional or attitudinal state, such as "Certainness" (certain, uncertain, neutral, mixed) and "Frustration/Anger" (frustrated/angry, non-frustrated/angry). These were manually annotated in their study.
They train and test predictive models using these parameters on their ITSPOKE corpora. For student learning, they specifically measure learning gains by controlling for pretest scores.
Datasets
The study utilizes three corpora collected from the ITSPOKE system:
- SYN03 (2003): Synthesized voice, used for an initial evaluation.
- PR05 (2005): Pre-recorded tutor voice.
- SYN05 (2005): Synthesized tutor voice.
These corpora consist of college students who had taken no prior physics. Data collection involved reading background material, taking a pretest, interacting with ITSPOKE for five problems (dialogues), and taking a posttest. User satisfaction surveys were administered at the end of the session for PR05 and SYN05. User affect annotations were performed on SYN03 and a subset of PR05.
Key Insights and Ideas
- Tutoring Systems are Different: The core hypothesis is that systems designed for learning, rather than just task completion (like booking flights), might require different performance metrics and parameters. User satisfaction might not be the most critical or easily predictable outcome.
- Generic Parameters Fall Short for Satisfaction: While system-generic parameters are standard in PARADISE, they were surprisingly weak predictors of user satisfaction in this tutoring system. This suggests that the nuances of a tutoring interaction are not fully captured by these broad metrics.
- Tutoring-Specifics Matter for Learning: Parameters directly related to the tutoring process, like student correctness, were crucial for predicting student learning. This highlights the importance of domain-specific features.
- User Affect Shows Promise for Learning: User affect parameters, though not used in prior PARADISE studies for satisfaction, demonstrated an ability to improve the predictive power of student learning models, suggesting affect plays a role in how well students learn.
- Model Sensitivity: User satisfaction models were highly sensitive to changes in training data, indicating potential overfitting and poor generalizability. Student learning models, while still sensitive, showed more robustness.
Key Results
- User Satisfaction Models: The models predicting user satisfaction had low R² values, both for training and testing, and showed little overlap in predictors across different corpora. For example, models trained on PR05 and tested on SYN05 yielded an R² of only 0.018. Even within-corpus models struggled to explain more than a quarter of the variance.
- Student Learning Models: These models were significantly more powerful. The first six models (using only system-generic and tutoring-specific parameters) explained over 50% of the POST variance in training data and often similarly in test data. For instance, a model trained on PR05:half1 achieved an R² of 0.443 for training and 0.079 for testing.
- Impact of User Affect: When user affect parameters were included, they improved the predictive power of one student learning model (SYN03) for both training and testing. For example, adding "Neutral Certainness" increased the R² for testing. However, the impact was not uniform across all models.
Limitations and Future Work
- User Satisfaction Metric Granularity: The authors suggest that collecting user satisfaction surveys only at the end of the entire tutoring session (5 dialogues) might be too coarse-grained. Future work could explore per-dialogue satisfaction measures.
- Goal Mismatch: Tutoring systems are primarily designed to maximize learning, not necessarily user satisfaction. This fundamental difference in design goals might explain why satisfaction is harder to predict.
- User Affect Annotation: Manual annotation of user affect is labor-intensive. Future work will focus on automatically predicting affect using acoustic-prosodic and lexical features, which have shown promise in other studies.
- Dialogue Acts: The authors plan to annotate dialogue acts and investigate their utility in prediction models, as these have been used in prior PARADISE applications.
- Linguistic Features: They also intend to explore linguistic features that reflect affective states as a potential replacement for direct affect annotation.
Summary updated