English Broadcast News Speech Recognition by Humans and Machines

Samuel Thomas, Masayuki Suzuki, Yinghui Huang, Gakuto Kurata, Zoltán Tüske, George Saon, Brian Kingsbury, Michael Picheny, Tom Dibert, Alice Kaiser-Schatzlein, Bern Samko

Abstract

With recent advances in deep learning, considerable attention has been given to achieving automatic speech recognition performance close to human performance on tasks like conversational telephone speech (CTS) recognition. In this paper we evaluate the usefulness of these proposed techniques on broadcast news (BN), a similar challenging task. We also perform a set of recognition measurements to understand how close the achieved automatic speech recognition results are to human performance on this task. On two publicly available BN test sets, DEV04F and RT04, our speech recognition system using LSTM and residual network based acoustic models with a combination of n-gram and neural network language models performs at 6.5% and 5.9% word error rate. By achieving new performance milestones on these test sets, our experiments show that techniques developed on other related tasks, like CTS, can be transferred to achieve similar performance. In contrast, the best measured human recognition performance on these test sets is much lower, at 3.6% and 2.8% respectively, indicating that there is still room for new techniques and improvements in this space, to reach human performance levels.

Introduction

In recent years, the speech recognition community has celebrated reaching "human parity" on conversational telephone speech (CTS). However, Broadcast News (BN) presents a distinct set of challenges, including wide-band signals, diverse background noise (music, field reports, studio settings), and a mix of prepared and spontaneous speech. Unlike CTS, which benefits from thousands of hours of high-quality transcriptions, BN data is often limited to "noisy" closed captions. This study investigates whether state-of-the-art techniques from the CTS domain can be successfully transferred to BN and quantifies the remaining gap between machine and human performance.

Historical trends in Word Error Rate (WER) across various speech recognition tasks.

Methodology

The researchers developed a sophisticated Automatic Speech Recognition (ASR) pipeline incorporating several advanced neural architectures:

Acoustic Modeling

Two distinct neural network types were employed to capture acoustic patterns:

  • Long Short-Term Memory (LSTM): A recurrent network with six bidirectional layers designed to capture long-range temporal dependencies in speech.
  • Residual Network (ResNet): A deep convolutional network with 25 layers, adapted from computer vision to process speech features effectively.

Language Modeling

A multi-stage approach was used to predict word sequences. Initial decoding utilized a standard n-gram model and a Feed-Forward Neural Network LM, followed by rescoring using two different LSTM-based Language Models to refine the output.

Training Strategy

To address the scarcity of high-quality labels, a "lightly supervised" approach was adopted. Existing models filtered 9,000 hours of closed-captioned data, selecting the most reliable 1,300 hours for training (BN-1300).

Evaluation and Human Benchmarking

The system was evaluated on two classic NIST benchmarks: DEV04F and RT04. To establish a rigorous baseline, the authors partnered with professional transcribers to process the same test sets, measuring human performance at 3.6% and 2.8% WER respectively.

Key Results and Discussion

The study established new state-of-the-art results for BN benchmarks, achieving a Word Error Rate (WER) of 6.5% on DEV04F and 5.9% on RT04. While these results represent significant progress, a persistent "human parity gap" remains.

Comparison of ASR and human performance by error type.

Analysis of error types reveals that machines struggle significantly more than humans with short "function words" (e.g., the, and, of). Humans leverage linguistic context and real-world knowledge to infer these words even in muffled audio, whereas machines rely more heavily on clear acoustic evidence. Additionally, humans are more diligent in marking hesitations, which contributes to their superior accuracy.

Conclusion

While techniques like ResNet acoustic models and LSTM language model rescoring generalize well from telephone speech to broadcast news, machines still lag behind professional transcribers by approximately 3% absolute WER. Future research should focus on improving the recognition of poorly articulated function words and better modeling of non-speech events to bridge this semantic gap.

Summary updated

Abstract

With recent advances in deep learning, considerable attention has been given to achieving automatic speech recognition performance close to human performance on tasks like conversational telephone speech (CTS) recognition. In this paper we evaluate the usefulness of these proposed techniques on broadcast news (BN), a similar challenging task. We also perform a set of recognition measurements to understand how close the achieved automatic speech recognition results are to human performance on this task. On two publicly available BN test sets, DEV04F and RT04, our speech recognition system using LSTM and residual network based acoustic models with a combination of n-gram and neural network language models performs at 6.5% and 5.9% word error rate. By achieving new performance milestones on these test sets, our experiments show that techniques developed on other related tasks, like CTS, can be transferred to achieve similar performance. In contrast, the best measured human recognition performance on these test sets is much lower, at 3.6% and 2.8% respectively, indicating that there is still room for new techniques and improvements in this space, to reach human performance levels.
BibTeX
@inproceedings{thomas2019english,
  author = {Samuel Thomas and Masayuki Suzuki and Yinghui Huang and Gakuto Kurata and Zoltán Tüske and George Saon and Brian Kingsbury and Michael Picheny and Tom Dibert and Alice Kaiser-Schatzlein and Bern Samko},
  title = {{English Broadcast News Speech Recognition by Humans and Machines}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=BnE4wb0A8g9},
}
Figure