TL;DR
This paper investigates whether Text-To-Speech (TTS) synthesis can lead to fewer errors in dictation exercises for foreign language learners compared to human-recorded audio. The study found that learners made significantly fewer errors when transcribing TTS-generated sentences, suggesting TTS might be a valuable tool for language learning despite not being specifically tuned for it.
What Problem Does This Paper Address?
Computer-Assisted Language Learning (CALL) systems often aim to automate the creation of learning materials, saving educators time and effort. A key component of this is generating audio content for exercises like dictation. While Text-To-Speech (TTS) synthesis offers a way to automatically generate this audio, its effectiveness and appropriateness for language learning, particularly in dictation tasks, remains debated. Previous studies have integrated TTS without rigorous evaluation, and those that have assessed it often found expressiveness lacking. This paper directly addresses this gap by conducting a controlled experiment to compare student performance in a dictation exercise using TTS-generated speech versus traditional pre-recorded human speech.
Methods
The researchers designed a dictation experiment using sentences from a European Portuguese (PSL) textbook. They created two sets of audio stimuli: one using a DIXI TTS engine (concatenative unit selection synthesis) with a single male voice ("Vicente"), and another using pre-recorded audio from the textbook's accompanying CD, featuring multiple male and female voices. Forty sentences were selected, with 20 in each set, carefully matched for lexical content and difficulty.
The experiment involved 11 ERASMUS students with intermediate Portuguese language skills. Each student completed a dictation exercise where they transcribed 40 sentences, half from the TTS set and half from the human-recorded set. The sentences were played three times each. Errors were manually categorized using a taxonomy that included spelling, addition, omission, and wrong-word errors (further subdivided into form, word, and contraction). Word Error Rate (WER) was the primary metric for evaluating performance.
Datasets
The study used sentences extracted from the "Português XXI" textbook, level A1, for European Portuguese learners.
- Synthesized Set: 20 sentences generated by the DIXI TTS engine.
- Pre-recorded Set: 20 sentences from the textbook's accompanying CD audio.
No new benchmark datasets were introduced; existing textbook materials were adapted.
Key Insights and Ideas
- TTS is surprisingly effective for dictation: The core finding is that learners performed better with TTS-generated speech, which was unexpected given the TTS voice wasn't optimized for learning. This challenges the assumption that human-recorded speech is always superior for this type of task.
- Speech rate matters: The slower speech rate of the TTS voice (15% slower) is hypothesized as a key factor contributing to the reduced error rate. This suggests that pacing, rather than just the source of the voice, is crucial for comprehension.
- Error patterns differ: While overall error rates were lower for TTS, the types of errors shifted. Specifically, fewer "wrong-word" errors (substitutions) occurred with TTS, indicating students might have been better at identifying the intended words. Conversely, function words were transcribed more accurately with TTS, while content words saw more errors, a reversal of the trend seen with human speech.
- Learners didn't detect the synthetic voice: Notably, students did not identify the TTS utterances as synthetic, even when specifically asked about the quality. This suggests TTS can be seamlessly integrated without negatively impacting the learning experience due to artificiality.
Key Results
- Global Word Error Rate (WER):
- Synthesized (TTS) speech: 26.6%
- Human-recorded speech: 33.9%
- This difference was statistically significant (p=0.0019).
- Function vs. Content Words:
- TTS: 46.9% WER for function words, 53.1% for content words.
- Human: 53.2% WER for function words, 46.8% for content words.
- Wrong-word errors: Lower in TTS (37.3%) compared to human (40.9%).
- Spelling errors: Higher in TTS (16.1%) compared to human (14.4%).
Limitations and Future Work
- Small sample size: The study involved only 11 students, which the authors acknowledge limits the generalizability of the findings.
- Single TTS voice: Only one male TTS voice was used. Future work should explore different voices and their impact.
- "Out of the box" TTS: The TTS engine was used without specific tuning for learning purposes. Further experiments could explore the impact of adjusting speech rate and pronunciation reduction parameters.
Future Work: * Repeat the experiment with a larger, more diverse group of students (varying proficiency levels and linguistic backgrounds). * Conduct controlled experiments to assess the impact of specific phonetic features and TTS parameters (like speech rate and co-articulation) on perception. * Investigate the potential of TTS for other CALL activities beyond dictation.
Summary updated