Abstract
We describe our work in building a spoken language interface for a tutorial dialogue system. Our goal is to allow natural, unrestricted student interaction with the computer tutor, which has been shown to improve the student's learning gain, but presents challenges for speech recognition and spoken language understanding. We discuss the choice of system components and present the results of development experiments in both acoustic and language modelling for speech recognition in this domain.
Introduction
Intelligent Tutoring Systems (ITS) like BEETLE II are designed to maximize learning gain by encouraging students to engage in "self-explanation." While typed tutoring systems have proven effective, spoken interaction is a more natural medium that can further enhance learning. However, transitioning from text to speech introduces significant technical hurdles. Students interacting with a tutor do not follow a rigid script; their speech is often unconstrained, filled with disfluencies (like "um" and "ah"), and contains complex technical terminology.
Researchers at the University of Edinburgh addressed these challenges by integrating a robust spoken language interface into BEETLE II. This system must handle unpredictable student responses in real-time, adapting to individual voices with very limited data.
System Architecture
The spoken language interface is integrated into the BEETLE II framework through a modular architecture. The system processes student input through an Automatic Speech Recognition (ASR) engine, followed by a parser and a semantic interpreter. The Dialogue Manager coordinates these inputs with a Knowledge Base to determine the appropriate pedagogical response. The output is then generated and delivered via a Text-to-Speech (TTS) engine.
Methodology
To bridge the gap between general speech and the specific domain of electricity tutoring, the team focused on two primary areas of adaptation:
Language Modeling (LM) Adaptation
Because spoken responses are more verbose and messy than typed ones, a model trained only on typed data (like the student09 corpus) is insufficient. The researchers employed unigram marginal adaptation. This technique adjusts the word probabilities of a large background model (trained on the Fisher Corpus) to match the vocabulary of a small set of domain-specific spoken data (adapt11), while preserving the higher-order n-gram relationships of the original model.
Acoustic Modeling (AM) and Online Adaptation
Lacking large-scale in-domain audio, the team used a base model trained on the 130-hour AMIDA corpus of meeting speech. To adapt to individual student voices in real-time, they used Constrained Maximum Likelihood Linear Regression (CMLLR). To prevent the model from becoming unstable when data is scarce (e.g., at the start of a session), they implemented count smoothing with an "identity prior." This ensures the system only makes significant updates to the acoustic model once enough evidence has been gathered.
Experimental Results
The adaptation strategies yielded significant improvements across both language and acoustic modeling:
- Perplexity Reduction: By interpolating general speech data with specific typed and spoken domain data, the model's perplexity (a measure of prediction difficulty) dropped from a baseline of 41.49 to 25.45.
- Word Error Rate (WER) Improvements: The baseline WER without adaptation was 48.8%. While standard full-covariance adaptation actually degraded performance to 53.7% due to data sparsity, the smoothed adaptation with a regression class tree successfully reduced the WER to 37.7%.
Conclusion
While a 37.7% WER remains a challenge for downstream semantic parsing, it represents a major advancement in handling spontaneous, unconstrained student speech. Future work will explore "lattice re-scoring," allowing the semantic parser to evaluate multiple hypotheses from the ASR engine to find the most scientifically plausible interpretation. Additionally, reinforcement learning may be used to further optimize tutoring strategies based on the evolving dialogue state.
Summary updated