TL;DR
This paper presents the early user testing of ARTUR, a novel computer-based speech training system designed to help children with speech and language disorders improve their articulation. The study highlights the usability of the interface, even for young children, but emphasizes the need for more engaging motivational elements and refined feedback mechanisms to enhance learning.
What Problem Does This Paper Address?
Learning to speak involves imitation and self-correction, processes that can be challenging for children with hearing impairments or language disorders. These children may struggle to perceive their own speech errors, making it difficult to achieve accurate pronunciation. Traditional speech therapy relies on visual cues, tactile sensations, and auditory feedback, but these can be time-consuming and difficult for therapists to provide consistently. Computer-based speech training (CBST) systems offer a potential solution by providing immediate, visual feedback. However, existing CBST systems often fall short in providing sufficient, personalized feedback for unsupervised training, particularly for individuals with limited auditory perception. This paper addresses the need for a more effective and engaging CBST system by evaluating the user interface of ARTUR, a system that aims to provide detailed articulatory feedback through 3D animations and clear instructions.
Methods
The core of this study involved a "Wizard of Oz" (WOZ) experiment. In this setup, a human "Wizard" (the first author) simulated the functions of the ARTUR system, specifically the mispronunciation detection and articulatory inversion components. This allowed the researchers to test and refine the human-computer interface and feedback display before fully developing the underlying speech technology.
The ARTUR system itself is designed with several key components:
- Virtual Tutor (Artur): A 3D animated character that provides feedback.
- Articulatory Feedback: Uses 3D animations of the face and vocal tract (tongue, jaw, palate) to show the difference between the user's pronunciation and the target.
- Acoustic to Articulatory Inversion: A complex process to recreate vocal tract movements from speech signals, enhanced by visual data from facial movements.
- Speech Recognition: Detects mispronounced speech by comparing user utterances to target sounds, considering potential error patterns based on language background and hearing impairment.
- User Interface: Designed to be intuitive, featuring the virtual tutor, training words, and interaction buttons.
The WOZ study involved two groups of children:
- Older Children (9-14): With extensive experience in speech training and CBST.
- Younger Children (6): With limited or no prior experience with CBST.
Participants were asked to practice specific sounds and words using the ARTUR interface. Following the training session, semi-structured interviews were conducted with the children and accompanying adults to gather feedback on the system's usability, clarity of feedback, and overall engagement.
Datasets
Not explicitly detailed in the paper. The study mentions that the WOZ setup involved collecting audio and video data that would be used as training material for the mispronunciation detection component in future iterations of the system. The articulatory models (face and vocal tract) were developed using Magnetic Resonance Imaging (MRI) data and supplemented with Electromagnetic Articulography (EMA) and Electro-palatography (EPG) measurements.
Key Insights and Ideas
- Usability for Children: The ARTUR interface proved to be usable by children without prior training or instructions, even for the younger group, which is a crucial step towards home-based use.
- Value of Detailed Feedback: Children, particularly those with more speech training experience, appreciated the specific instructions on how to improve articulation (e.g., "Try to retract the tongue tip"). This detailed, actionable feedback was seen as a significant advantage over other CBST systems.
- Motivational Gap: While the system was generally well-received, a significant insight was the need for more motivational factors. Children, especially the younger ones, found the current interface lacking in engagement, suggesting that game-like features are essential for sustained practice.
- Visual Feedback Challenges: The representation of the hard palate in the 3D models was consistently misunderstood by children, indicating a need for clearer visual design.
- Interaction Button Utility: The interaction buttons (e.g., "Show word," "Slow," "Show difference") were underutilized during the training sessions, likely due to the children's inability to read the text labels and the system's design (buttons disabling during speech).
- The "Wizard of Oz" Advantage: The WOZ approach was invaluable for collecting rich qualitative data and understanding user needs before committing to complex technological development. It allowed for flexible feedback and immediate refinement of the interaction.
Key Results
- Positive Reception: Both older and younger children generally responded positively to ARTUR, describing it as "intelligent" and "good."
- Preference for ARTUR: Older children explicitly preferred ARTUR over other CBST systems like SpeechViewer and Box-of-Tricks, primarily due to its corrective instructions.
- Mixed Animation Reception: The 3D animations of speech organs received mixed reviews; some found them "really good," while others found them "strange" or noted technical flaws.
- Feedback Clarity: While the general concept of feedback was understood, the specific representation of the hard palate was confusing.
- Limited Practice: The limited number of sounds and words available for practice was a drawback for the older children who had already established some contrasts.
- "Relaxed" Practice: Children found practicing with ARTUR to be a "relaxed" experience compared to practicing with a speech therapist, highlighting the benefit of independent practice.
Limitations and Future Work
Limitations: * Small Sample Size: The study involved a small number of participants (two groups of three children each, plus one adult learner). * Short Training Duration: The training sessions were brief (approximately 10 minutes), limiting the ability to observe significant longitudinal improvements or gather extensive quantitative data. * Wizard of Oz Simulation: The mispronunciation detection and articulatory inversion were simulated by a human, meaning the system's actual performance in these areas was not tested. * Crude Feedback Matrix: The 10 feedback options available to the Wizard were sometimes too general or too detailed, making it difficult to address subtle errors.
Future Work: * Motivational Features: The authors strongly advocate for incorporating more game-like features and varied encouragement strategies to increase user motivation and engagement, especially for younger children. * Improved Visual Feedback: Re-implementing the palate display and potentially other visual elements to be clearer and more intuitive. * Adaptive Feedback: Developing a system where feedback detail adapts to the user's performance and the confidence score of the error detection. * More Granular Feedback: Supplementing the current feedback options with a more nuanced system that can handle uncertainty and less obvious misclassifications. * Focus on Specific Articulations: Ensuring the training session clearly defines its focus (e.g., a specific articulation) and provides feedback accordingly, while logging other errors for future sessions. * User Control: Allowing users more control over when feedback is given, potentially with options to skip feedback or request specific types of guidance. * Longitudinal Studies: Conducting studies with larger datasets and longer training periods to quantitatively evaluate ARTUR's effectiveness in improving speech production.
Summary updated