TL;DR
This paper introduces novel methods for automatically analyzing child speech therapy sessions by integrating ultrasound tongue imaging with audio data. By leveraging visual biofeedback, they improve speaker diarization (identifying who is speaking) and word alignment (pinpointing when specific words are spoken), offering significant potential to reduce manual annotation effort for therapists.
What Problem Does This Paper Address?
Children with developmental speech sound disorders (SSDs) often require speech therapy. While visual biofeedback (VBF), particularly ultrasound tongue imaging (UTI), has shown promise in helping these children improve their speech production, the analysis of therapy sessions is currently a laborious manual process. Therapists spend considerable time transcribing and annotating recordings to track progress. This paper aims to automate two key aspects of this analysis: speaker diarization (distinguishing between the child and the therapist) and time-alignment of target words. The challenges are amplified by the nature of child speech (e.g., disfluencies, mispronunciations) and the lack of full transcriptions, making traditional speech processing techniques insufficient.
Methods
The researchers propose two main novel approaches:
- Speaker Diarization: They introduce "Estimated Tongue Activity" (ETA), a time-domain signal derived from the variance of ultrasound frames. High variance in ETA indicates significant tongue movement, which they hypothesize correlates with child speech, while low variance suggests therapist speech. This ETA signal is used to complement traditional voice activity detection (VAD) for diarization.
- Word Alignment: To improve the accuracy of aligning spoken words with a given prompt, they augment standard acoustic models with ultrasound embeddings. These embeddings are generated by a Convolutional Neural Network (CNN) architecture designed to extract meaningful representations from raw ultrasound frames. The CNN takes sequences of ultrasound frames as input and outputs low-dimensional representations that capture articulatory information. These ultrasound embeddings are then concatenated with audio features (like MFCCs and pitch) to train more robust acoustic models.
Datasets
The primary dataset used is UltraSuite [17], a repository of synchronized ultrasound and audio data from child speech therapy sessions. It includes:
- Ultrax Typically Developing (UXTD): Recordings from 58 typically developing children.
- Ultrax Speech Sound Disorders (UXSSD): Recordings from 8 children with speech sound disorders.
- Ultraphonix (UPX): Recordings from 20 children with speech sound disorders.
Additionally, the PF-STAR children's speech corpus [24] was used for training Universal Background Models (UBMs) for i-vector systems and for bootstrapping HMM/GMM models, providing out-of-domain data.
Key Insights and Ideas
- Ultrasound as a Complementary Signal: The core idea is that ultrasound provides articulatory information not present in audio alone, and this visual data can significantly enhance speech processing tasks in this specific domain.
- ETA for Child Speech Detection: The Estimated Tongue Activity (ETA) signal offers a simple yet effective way to differentiate child speech from therapist speech, leveraging the inherent differences in their articulatory dynamics.
- CNNs for Articulatory Feature Extraction: Using CNNs to learn embeddings from raw ultrasound data is a powerful way to distill complex visual information into a format usable by acoustic models.
- Combined Models for Optimal Performance: The best results for word alignment were achieved by combining models trained on out-of-domain audio data (PF-STAR) with models augmented by ultrasound embeddings. This highlights the synergistic benefit of multiple data sources.
Key Results
- Speaker Diarization: Systems augmented with ETA outperformed VAD-only baselines, especially for disordered speech where therapist intervention is more frequent. For disordered speech, the best system achieved a Diarization Error Rate (DER) of 24.0% and a Confusion Error (Conf) of 8.4%, significantly better than VAD alone (DER 47.6%, Conf 29.7%).
- Word Alignment:
- For typically developing children, combining out-of-domain audio data (USuite+PFstar) and ultrasound embeddings (USuite+UTI) through model interpolation (with a weight of $\alpha=0.7$ for the audio-augmented model) yielded the lowest Word Error Rate (WER) of 28.68%.
- For disordered speech, the same combination strategy (with $\alpha=0.7$) achieved the lowest WER of 58.14%.
- The paper notes that while ultrasound provided improvements, augmenting with out-of-domain audio data had a more substantial impact on recognition accuracy.
Limitations and Future Work
- Limited Annotated Data: A significant challenge is the scarcity of manually annotated training data for child speech therapy sessions, which impacts the robustness of diarization systems.
- Speaker/Session Adaptation: The current models do not adapt to individual speaker-specific pronunciations or track changes in pronunciation over the course of therapy. Future work could explore learning these variations.
- Handling Insertions/Deletions: The alignment system could be improved by relaxing constraints to better handle spoken words that deviate from the prompt.
- Therapist Input for Adaptation: The authors suggest that diarization systems could be improved by incorporating user input from therapists, especially since therapy sessions typically involve only two speakers.
- Longitudinal Learning: The temporal dependency of therapy sessions could be leveraged for longitudinal online learning.
Summary updated