Evaluation of state-of-the-art ASR Models in Child-Adult Interactions

Aditya Ashvin, Rimita Lahiri, Aditya Kommineni, Somer Bishop, Catherine Lord, Sudarsana Reddy Kadiri, Shrikanth Narayanan

Abstract

The ability to reliably transcribe child-adult conversations in a clinical setting is valuable for diagnosis and understanding of numerous developmental disorders such as Autism Spectrum Disorder (ASD). Recent advances in deep learning architectures and availability of large scale transcribed data has led to development of speech foundation models that have shown dramatic improvements in ASR performance. However, the ability of these models to translate well to conversational child-adult interactions is under studied. In this work, we provide a comprehensive evaluation of ASR performance on a dataset containing child-adult interactions from autism diagnostic sessions, using Whisper, Wav2Vec2, HuBERT, and WavLM. We find that speech foundation models show a noticeable performance drop (15-20% absolute WER) for child speech compared to adult speech in the conversational setting. Then, we employ LoRA on the best performing zero shot model (whisper-large) to probe the effectiveness of fine-tuning in a low resource setting, resulting in ~8% absolute WER improvement for child speech and ~13% absolute WER improvement for adult speech.

Introduction

Automatic Speech Recognition (ASR) has reached near-human parity for adult speech in clean, controlled conditions. However, child speech remains a significant challenge due to physiological differences—such as shorter vocal tracts and higher fundamental frequencies—and a chronic lack of high-quality labeled training data.

This challenge is particularly acute in the context of Autism Spectrum Disorder (ASD) diagnostics. Sessions like the Autism Diagnostic Observation Schedule (ADOS) involve spontaneous, often fragmented, and atypical conversational patterns between a clinician and a child. Reliable transcription of these sessions is vital for clinical research and quantifying symptom severity, yet most foundation models are trained on adult-centric datasets. This study evaluates how state-of-the-art foundation models handle this specific, high-stakes conversational environment.

Methodology and Data

Experimental Pipeline

The research followed a two-phase approach: zero-shot evaluation followed by parameter-efficient fine-tuning.

Overview of the ASR evaluation and fine-tuning pipeline for child-adult interactions.
  1. Zero-shot Evaluation: Several prominent speech foundation models were tested without additional training, including supervised models (Whisper Base, Medium, and Large-v3) and Self-Supervised Learning (SSL) models (Wav2Vec 2.0, HuBERT, and WavLM).
  2. Parameter-Efficient Fine-Tuning (PEFT): Using the best-performing zero-shot model (Whisper Large-v3), the team applied Low-Rank Adaptation (LoRA). This allowed for adapting the massive model by training only a tiny fraction of its parameters, which is ideal for the low-resource nature of clinical child speech data.

The ADOSMod3 Dataset

The study utilized a specialized dataset called ADOSMod3, consisting of 165 video-recorded ADOS sessions from two clinical sites. The dataset includes approximately 17 hours of audio (8 hours child, 9 hours adult) comprising 115,111 utterances. The children involved ranged from 3.5 to 13 years old, with clinical diagnoses including ASD, ADHD, and various language or mood disorders. All sessions were manually transcribed following SALT (Systematic Analysis of Language Transcripts) guidelines.

Performance Analysis and Key Insights

The "Child Gap"

Even the most advanced models exhibited a massive performance disparity between speaker types. In zero-shot tests, child speech consistently had a Word Error Rate (WER) roughly 30% higher than adult speech. Whisper models, trained on 680,000 hours of labeled data, significantly outperformed SSL models like WavLM and HuBERT, suggesting that the diversity of Whisper's training data aids generalization in "messy" clinical environments.

The Impact of Utterance Length

A critical finding was the strong correlation between utterance length and error rate. Single-word or two-word responses—which are common in ASD diagnostic play—suffered from the highest WER.

Average Word Error Rate (WER) as a function of transcription length for child and adult speech.

As shown in the figure above, the WER for both groups decreases as the number of words increases. However, the gap remains persistent. Because models like Whisper are often optimized for 30-second audio chunks, they likely struggle with the lack of acoustic and linguistic context in very brief snippets.

Channel Adaptation

Interestingly, fine-tuning on adult-only speech improved child ASR performance, and vice versa. This suggests that the model is not only learning speaker-specific features but is also adapting to the "channel"—the specific acoustic environment and recording conditions of the clinical sessions.

Results and Conclusion

Whisper Large-v3 emerged as the top zero-shot performer, achieving a 55.78% WER for children and 25.96% WER for adults. Applying LoRA fine-tuning to this model brought the child WER down to 46.01% (a ~10% relative improvement) and the adult WER down to 13.09% (a ~50% relative improvement).

While fine-tuning yielded significant gains, a 46% WER for child speech remains too high for fully automated clinical use without human oversight. Future work will focus on data augmentation strategies (such as pitch shifting) to simulate more child-like speech and investigating methods to better handle very short utterances by modifying the standard windowing approaches used in foundation models.

Summary updated

Abstract

The ability to reliably transcribe child-adult conversations in a clinical setting is valuable for diagnosis and understanding of numerous developmental disorders such as Autism Spectrum Disorder. Recent advances in deep learning architectures and availability of large scale transcribed data has led to development of speech foundation models that have shown dramatic improvements in ASR performance. However, the ability of these models to translate well to conversational child-adult interactions is under studied. In this work, we provide a comprehensive evaluation of ASR performance on a dataset containing child-adult interactions from autism diagnostic sessions, using Whisper, Wav2Vec2, HuBERT, and WavLM. We find that speech foundation models show a noticeable performance drop (15-20% absolute WER) for child speech compared to adult speech in the conversational setting. Then, we employ LoRA on the best performing zero shot model (whisper-large) to probe the effectiveness of fine-tuning in a low resource setting, resulting in ~8% absolute WER improvement for child speech and ~13% absolute WER improvement for adult speech.
BibTeX
@misc{ashvin2024evaluation,
  author = {Aditya Ashvin and Rimita Lahiri and Aditya Kommineni and Somer Bishop and Catherine Lord and Sudarsana Reddy Kadiri and Shrikanth Narayanan},
  title = {{Evaluation of state-of-the-art ASR Models in Child-Adult Interactions}},
  year = {2024},
  eprint = {2409.16135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.16135},
}
Figure