TL;DR
This paper introduces ITSPOKE, a spoken dialogue system that enhances a text-based physics tutor (Why2-Atlas) by incorporating speech. The goal is to understand how adding spoken interaction impacts student learning and to explore the use of acoustic-prosodic features for better pedagogical insights.
What Problem Does This Paper Address?
The paper addresses the challenge of improving the effectiveness of intelligent tutoring systems (ITS). While many ITS have shown success, most are text-based. The authors hypothesize that spoken dialogue could be more effective, drawing parallels to human tutoring where spontaneous self-explanation, which occurs more frequently in speech, leads to better learning. They also explore the potential of using speech's acoustic and prosodic features (like tone of voice) to understand student emotions, which could further enhance tutoring. The core problem is evaluating the impact and utility of integrating spoken dialogue capabilities into existing text-based tutoring systems.
Methods
ITSPOKE is built upon the Why2-Atlas conceptual physics tutoring system. The core innovation is the integration of speech recognition and synthesis. Student speech is captured by a microphone, digitized, and sent to the Sphinx2 automatic speech recognizer. The recognized text is then fed into Why2-Atlas for analysis. The tutor's responses, generated by Why2-Atlas, are converted to speech using the Cepstral text-to-speech system. ITSPOKE acts as a proxy server, managing the communication between the Why2-Atlas server and the client, deciding when to listen and when to speak. The system is implemented in Python, with Sphinx2 and Cepstral integrated via APIs.
Datasets
The paper mentions that the stochastic language models for speech recognition were initially trained using 4551 typed student utterances from a previous Why2-Atlas evaluation. These models were later enhanced with spoken utterances collected during ITSPOKE's pilot testing. The performance analysis section describes an ongoing formal evaluation comparing ITSPOKE with other tutoring methods, which began in November 2003. As of March 2004, 80 dialogues from 16 students had been collected, totaling 21 hours of speech.
Key Insights and Ideas
- Speech enhances spontaneous explanation: The authors hypothesize that spoken dialogue encourages students to explain themselves more freely, mirroring effective human tutoring strategies.
- Acoustic-prosodic features offer pedagogical value: Beyond just the words spoken, the way a student speaks (their tone, rhythm, etc.) can provide rich information about their emotional state, which could be leveraged for more adaptive tutoring.
- Bridging the gap between human and computer tutors: ITSPOKE aims to bring the benefits of natural, spoken interaction, often seen in human tutoring, into the realm of computer-based education.
- Empirical evaluation is crucial: The paper emphasizes the need for rigorous, empirical studies to understand the actual impact of spoken dialogue on learning gains, efficiency, and usability.
Key Results
The paper reports on the ongoing evaluation. As of March 2004, an average dialogue contained 21.3 student turns and 26.3 tutor turns, with a mean student turn length of 2.8 words. ITSPOKE's current Word Error Rate (WER) for speech recognition is 31.2%. However, the authors note that semantic accuracy, rather than transcription accuracy, is more important for dialogue evaluation. They found that semantic analysis based on speech recognition is the same as based on perfect transcription 92% of the time. The system also handles rejection prompts (asking the user to repeat) and timeout prompts (repeating the question) effectively.
Limitations and Future Work
The authors acknowledge that the current Word Error Rate of 31.2% is a limitation, although they mitigate this by focusing on semantic accuracy. They also note that the system is still under development and undergoing formal evaluation. Future work includes analyzing the collected corpus to quantify performance improvements, investigating the use of acoustic-prosodic features for predicting student emotions, modifying the dialogue manager to incorporate speech-optimized tutorial strategies, and enhancing ITSPOKE to predict and adapt to student emotion. They plan to build upon previous work in adaptive dialogue systems, where adaptation to problematic situations measurably improved system performance.
Summary updated