Evaluating Automatic Speech Recognition Pipelines for Mandarin-English Bilingual Child Language Assessment in Telehealth

Hongchen Wu, Yao Du, Zirong Li, Yixin Gu, D. Jayaprakash, Sheng Li2025 · DOI

TL;DR

This paper tackles the challenge of accurately transcribing Mandarin-English bilingual children's speech in telehealth settings, a crucial step for diagnosing language delays. By evaluating various Automatic Speech Recognition (ASR) pipelines, they found that integrating speaker diarization with OpenAI's Whisper model significantly improves accuracy, reducing word error rates for child speech and code-switching. This work establishes a benchmark for bilingual child speech technology and offers a practical framework for enhancing telehealth-based language assessments.

What Problem Does This Paper Address?

Bilingualism is on the rise, but assessing bilingual children's language development, especially in telehealth, presents significant hurdles. These include a shortage of bilingual clinicians and the time-consuming nature of manually annotating speech data. This manual annotation is critical for identifying language delays and disorders, but it's prone to errors and delays, potentially leading to misdiagnoses. The complexity is amplified by factors like code-switching (alternating languages within a conversation), children's variable speech patterns, and the presence of multiple speakers (clinician, child, parent) in telehealth sessions. Existing ASR models often struggle with these combined challenges, necessitating research into robust pipelines for this specific domain.

Methods

The researchers explored six different ASR pipelines, primarily focusing on integrating OpenAI's Whisper model with various preprocessing techniques. The core idea was to find an easy-to-implement and cost-effective solution for clinicians. The methods can be broadly categorized:

  • Single-Pass Whisper: Directly feeding audio to Whisper's small and medium models without prior processing.
  • Two-Pass Language-Aware Approach: Splitting audio, using a "tiny" Whisper model for language detection, and then using a larger model for transcription. A fallback mechanism using character distributions was employed when language detection failed.
  • Overlapped Audio Segmentation: Segmenting audio into overlapping chunks to preserve context, followed by transcription.
  • Snippet of Language-Aware: Using a short snippet for language detection before full transcription.
  • Pyannote Speaker Diarization + Whisper/Azure Speech: This was a key focus. Pyannote was used to identify individual speakers, and then Whisper or Azure Speech was used to transcribe each speaker's segment. This approach aimed to disentangle speakers and languages effectively.

The novelty lies in the systematic evaluation of these cascaded systems specifically for the challenging domain of bilingual Mandarin-English child telehealth speech, a combination not extensively studied before.

Datasets

The study utilized a custom dataset comprising 53 telehealth sessions, totaling approximately 20 hours of audio. This dataset was collected during Mandarin-English receptive language assessments for children aged 4-8 using the Mandarin-English Receptive Language Screener (MERLS). The data captures naturalistic parent-child interactions, including code-switching and varying speech quality, making it a valuable benchmark for evaluating ASR performance in this specific clinical context.

Key Insights and Ideas

  • Preprocessing is Crucial: Simple pre-processing steps, especially speaker diarization, dramatically improve ASR performance on this complex dataset, highlighting that raw audio fed directly to models is insufficient.
  • Whisper's Strength: OpenAI's Whisper, particularly the medium model, shows significant promise, often outperforming other models when combined with effective preprocessing.
  • Model Size Isn't Everything: Simply using larger Whisper models (medium vs. small) doesn't always guarantee better results; the pipeline's architecture and preprocessing play a more significant role.
  • Code-Switching and Child Speech are Hard: The data clearly shows that ASR models struggle more with code-switched utterances and children's speech, leading to higher word error rates.
  • Hallucinations and Errors: ASR models can generate "hallucinated" transcriptions in silent segments and misinterpret code-switching, indicating a need for domain-specific training and better handling of linguistic nuances.

Key Results

  • Top Performing Pipeline: The combination of Pyannote Speaker Diarization with Whisper medium (Method 5) achieved the highest overall accuracy, with an impressive 85.57% Word Accuracy.
  • Reduced Word Error Rates: This top pipeline reduced word error rates to approximately 35% for child speech and 30% for code-switching, rivaling fine-tuned transformer models.
  • Speaker Diarization Advantage: Methods incorporating Pyannote Speaker Diarization consistently outperformed other preprocessing techniques, demonstrating its effectiveness in separating speakers.
  • Quantitative Performance: Table 1 shows significant variations in performance across pipelines, with Method 5-medium leading in most metrics like Onset Time, Timestamp, Word, Language ID, and Speaker Diarization accuracy. Table 2 further details speaker- and language-specific word transcription accuracy.

Limitations and Future Work

Limitations:

  • The study acknowledges that the dataset, while valuable, is specific to Mandarin-English and MERLS tasks.
  • The authors note that ASR models can still struggle with audio quality issues (background noise, internet dropouts) and misidentify languages in short or unclear utterances.
  • Hallucinated transcriptions in silent segments and over-translation of code-switched sentences remain challenges.

Future Work:

  • Expanding the dataset to include more bilingual combinations and diverse assessment tasks.
  • Exploring other ASR models, including fine-tuning existing ones.
  • Investigating more advanced audio preprocessing techniques to further mitigate noise and quality issues.
  • Developing more robust methods for handling code-switching and children's speech variability.
  • The authors suggest that their findings offer a methodological framework for expanding clinical speech services to bilingual children.

Summary updated

BibTeX
@misc{wu2025evaluating,
  author = {Hongchen Wu and Yao Du and Zirong Li and Yixin Gu and D. Jayaprakash and Sheng Li},
  title = {{Evaluating Automatic Speech Recognition Pipelines for Mandarin-English Bilingual Child Language Assessment in Telehealth}},
  year = {2025},
  doi = {10.21437/interspeech.2025-2430},
}
Figure