A comparison of human and computer recognition accuracy for children's speech

Shona D'Arcy, Martin RussellInterspeech 2005

TL;DR

This paper investigates why automatic speech recognition (ASR) systems struggle with children's speech. By comparing human and machine performance on the same dataset, they found that both systems are similarly affected by factors like age and reduced audio bandwidth, suggesting these issues stem from inherent properties of children's speech rather than ASR technology limitations.

What Problem Does This Paper Address?

Recognizing children's speech with ASR systems is notoriously difficult, with error rates significantly higher than for adult speech. Previous research has shown that ASR performance degrades as children get younger and is more sensitive to reduced audio quality (like lower bandwidth). However, it was unclear whether these observed effects were artifacts of the ASR technology itself or if they reflected genuine challenges in perceiving children's speech. This paper aims to bridge this gap by directly comparing human and machine recognition performance on the same children's speech data to understand the root causes of these recognition difficulties.

Methods

The study employed a two-pronged approach:

  • Automatic Speech Recognition (ASR): They used a Hidden Markov Model (HMM) toolkit (HTK) to build bandwidth-dependent, tied-state decision tree triphone HMMs. These models were trained on a subset of the "PF_STAR" British English children's speech corpus. Crucially, they experimented with different audio bandwidths (11 kHz down-sampled to 8, 6, 4, and 3 kHz) to assess the impact of audio quality. They also used two test sets: a full test set and a smaller subset that directly matched the data used for human listeners, enabling a direct comparison.
  • Human Speech Recognition: Forty native English speakers, aged 14-50, participated. They listened to utterances from the children's speech corpus (at various bandwidths and from different age groups) and transcribed what they heard. Each utterance was played only once to simulate a more challenging listening scenario and to potentially reveal age and bandwidth effects more clearly.

Datasets

The primary dataset used was a subset of the British English children's speech data from the EU FP5 'PF_STAR' project. This corpus consists of over 852 minutes of read speech from children aged 4 to 15 years, transcribed at the word level. For this study, speech data from children aged 6 to 11 years was used, with original bandwidths of 11 kHz being down-sampled to 8, 6, 4, and 3 kHz.

Key Insights and Ideas

  • Children's Speech is Inherently Challenging: The fact that human listeners also exhibited significant error rates, particularly at lower bandwidths, strongly suggests that children's speech possesses acoustic properties that make it inherently difficult to understand, regardless of the recognition system.
  • Age and Bandwidth Effects Mirror Each Other: Both human and automatic systems showed similar trends: recognition accuracy decreased with age (younger children were harder to understand) and with reduced audio bandwidth. This consistency is a key finding.
  • Bandwidth Sensitivity is Pronounced for Younger Children: The study reinforces previous findings that children's speech, especially from younger speakers, contains crucial information at higher frequencies. Reducing bandwidth disproportionately impacts the intelligibility of their speech.
  • Human Performance is Superior, but Not Perfect: While human listeners performed vastly better than the ASR system, their error rates were still higher than typically expected for adult speech, further highlighting the inherent difficulty.

Key Results

  • ASR Performance: For 11-year-old children at full bandwidth (11 kHz), the word error rate for the ASR system was a staggering 73.39%. Error rates generally increased as age decreased and bandwidth decreased.
  • Human Performance: Human listeners achieved an average word error rate of 3.6% at 11 kHz bandwidth. While much lower than ASR, this is still a notable error rate.
  • Bandwidth Impact: For human listeners, error rates increased significantly as bandwidth dropped from 6 kHz to 4 kHz (31% increase) and from 4 kHz to 3 kHz (94% increase).
  • Age Impact: For human listeners, word error rates generally increased as the age group of the child decreased, with the exception of 11-year-olds, which the authors attribute to sample size.

Limitations and Future Work

  • Sample Size for 11-Year-Olds: The authors note that the higher error rate for 11-year-old children in the human recognition experiment is difficult to explain and might be due to the relatively small sample size for this age group.
  • Age as an Indicator: The paper acknowledges that "age" is a convenient but broad indicator of various factors (physical size, reading ability, pronunciation) that influence speech variability. Future work could explore these specific factors more directly.
  • Age-Independent ASR Models: The ASR system used "age-independent" HMMs trained on a broad age range. While this is a common approach, it might have contributed to the observed performance for older children. Future research could investigate age-specific ASR models.
  • Further Exploration of Acoustic Properties: The findings suggest that the acoustic properties of children's speech are the primary cause of recognition errors. Future work could delve deeper into characterizing these specific acoustic differences compared to adult speech.

Summary updated

BibTeX
@inproceedings{darcy2005comparison,
  author = {Shona D'Arcy and Martin Russell},
  title = {{A comparison of human and computer recognition accuracy for children's speech}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure