Abstract
With recent advances in deep learning, considerable attention has been given to achieving automatic speech recognition performance close to human performance on tasks like conversational telephone speech (CTS) recognition. In this paper we evaluate the usefulness of these proposed techniques on broadcast news (BN), a similar challenging task. We also perform a set of recognition measurements to understand how close the achieved automatic speech recognition results are to human performance on this task. On two publicly available BN test sets, DEV04F and RT04, our speech recognition system using LSTM and residual network based acoustic models with a combination of n-gram and neural network language models performs at 6.5% and 5.9% word error rate. By achieving new performance milestones on these test sets, our experiments show that techniques developed on other related tasks, like CTS, can be transferred to achieve similar performance. In contrast, the best measured human recognition performance on these test sets is much lower, at 3.6% and 2.8% respectively, indicating that there is still room for new techniques and improvements in this space, to reach human performance levels.
Introduction
In recent years, the speech recognition community has celebrated reaching "human parity" on conversational telephone speech (CTS). However, Broadcast News (BN) presents a distinct set of challenges, including wide-band signals, diverse background noise (music, field reports, studio settings), and a mix of prepared and spontaneous speech. Unlike CTS, which benefits from thousands of hours of high-quality transcriptions, BN data is often limited to "noisy" closed captions. This study investigates whether state-of-the-art techniques from the CTS domain can be successfully transferred to BN and quantifies the remaining gap between machine and human performance.
Methodology
The researchers developed a sophisticated Automatic Speech Recognition (ASR) pipeline incorporating several advanced neural architectures:
Acoustic Modeling
Two distinct neural network types were employed to capture acoustic patterns:
- Long Short-Term Memory (LSTM): A recurrent network with six bidirectional layers designed to capture long-range temporal dependencies in speech.
- Residual Network (ResNet): A deep convolutional network with 25 layers, adapted from computer vision to process speech features effectively.
Language Modeling
A multi-stage approach was used to predict word sequences. Initial decoding utilized a standard n-gram model and a Feed-Forward Neural Network LM, followed by rescoring using two different LSTM-based Language Models to refine the output.
Training Strategy
To address the scarcity of high-quality labels, a "lightly supervised" approach was adopted. Existing models filtered 9,000 hours of closed-captioned data, selecting the most reliable 1,300 hours for training (BN-1300).
Evaluation and Human Benchmarking
The system was evaluated on two classic NIST benchmarks: DEV04F and RT04. To establish a rigorous baseline, the authors partnered with professional transcribers to process the same test sets, measuring human performance at 3.6% and 2.8% WER respectively.
Key Results and Discussion
The study established new state-of-the-art results for BN benchmarks, achieving a Word Error Rate (WER) of 6.5% on DEV04F and 5.9% on RT04. While these results represent significant progress, a persistent "human parity gap" remains.
Analysis of error types reveals that machines struggle significantly more than humans with short "function words" (e.g., the, and, of). Humans leverage linguistic context and real-world knowledge to infer these words even in muffled audio, whereas machines rely more heavily on clear acoustic evidence. Additionally, humans are more diligent in marking hesitations, which contributes to their superior accuracy.
Conclusion
While techniques like ResNet acoustic models and LSTM language model rescoring generalize well from telephone speech to broadcast news, machines still lag behind professional transcribers by approximately 3% absolute WER. Future research should focus on improving the recognition of poorly articulated function words and better modeling of non-speech events to bridge this semantic gap.
Summary updated