English Broadcast News Speech Recognition by Humans and Machines

Samuel Thomas, Masayuki Suzuki, Yinghui Huang, Gakuto Kurata, Zoltán Tüske, George Saon, Brian Kingsbury, Michael Picheny, Tom Dibert, Alice Kaiser-Schatzlein, Bern Samko

TL;DR

This paper evaluates state-of-the-art deep learning techniques—originally perfected for conversational telephone speech—on the task of English Broadcast News (BN) recognition. While the researchers achieved new performance milestones for machines (5.9%–6.5% word error rate), they found a persistent "human parity gap," as professional human transcribers still outperform the best models by about 3% absolute.

What Problem Does This Paper Address?

In recent years, the speech recognition community has celebrated reaching "human parity" on conversational telephone speech (CTS). However, Broadcast News (BN) presents a different set of challenges: wide-band signals, diverse background noise (music, field reports, studio settings), and a mix of prepared and spontaneous speech.

A major hurdle in BN is the lack of high-quality training data. While CTS has thousands of hours of meticulously transcribed audio, BN only has about 140 hours of "gold standard" transcriptions. The rest of the available data (thousands of hours) relies on "noisy" closed captions. This paper investigates whether techniques that worked for CTS can be successfully transferred to the BN domain and exactly how far machines are from matching human accuracy in this specific context.

Methods

The authors built a sophisticated ASR (Automatic Speech Recognition) pipeline using a combination of several advanced techniques:

  • Acoustic Modeling: They employed two distinct types of neural networks to capture audio patterns:
  • LSTM (Long Short-Term Memory): A recurrent network with six bidirectional layers, excellent at capturing temporal dependencies in speech.
  • ResNet (Residual Network): A deep convolutional network with 25 layers, adapted from computer vision to process speech features.
  • Language Modeling: To predict word sequences, they used a multi-stage approach. They started with a standard n-gram model and a Feed-Forward Neural Network LM, then performed "rescoring" using two different LSTM-based Language Models.
  • Training Strategy: Because high-quality labels are scarce, they used a "lightly supervised" approach. They used existing models to filter 9,000 hours of closed-captioned data, selecting only the most reliable segments to create a 1,300-hour training set.

Datasets

  • Training: The BN-400 (430 hours) and BN-1300 (1,300 hours) corpora, derived from DARPA EARS and GALE programs.
  • Evaluation: Two classic NIST benchmarks: DEV04F and RT04.
  • Human Benchmarking: The authors partnered with Appen to have professional transcribers process the same test sets to establish a reliable "human" baseline.

Key Insights and Ideas

  • Cross-Domain Generalization: Techniques like ResNet acoustic models and LSTM language model rescoring, which were game-changers for telephone speech, transfer very effectively to broadcast news.
  • The "Function Word" Failure: Machines struggle significantly more than humans with short "function words" (e.g., the, and, of, a, that). Humans can often infer these words from context even if the audio is muffled, whereas machines rely more heavily on clear acoustic evidence.
  • The Importance of Hesitation: Humans are very diligent about marking hesitations (like "%hesitation"). While these don't necessarily change the "meaning" for a machine, they are crucial for human-level transcription accuracy and flow.
  • The Parity Gap: Despite the sophisticated models, humans are still significantly better at BN than machines. The human error rate is roughly half that of the machine error rate.

Key Results

The study established new state-of-the-art results for these specific BN benchmarks:

Test Set Machine WER (Best) Human WER
DEV04F 6.5% 3.6%
RT04 5.9% 2.8%

The machine's performance improved significantly when combining the LSTM and ResNet models, though the gain from model fusion was smaller on BN than what has been observed in the CTS domain.

Limitations and Future Work

The primary limitation is the semantic gap. Humans use their understanding of syntax and real-world knowledge to "fill in the blanks" for poorly articulated words, a feat current ASR systems still struggle to replicate.

The authors suggest that future work should focus on:

  1. Improving the recognition of short, poorly articulated function words.
  2. Better modeling of "hesitation" and non-speech events to match the nuance of human transcription.
  3. Closing the ~3% gap through even more advanced neural architectures or better utilization of the massive amounts of unlabelled broadcast data.

Summary updated

Abstract

With recent advances in deep learning, considerable attention has been given to achieving automatic speech recognition performance close to human performance on tasks like conversational telephone speech (CTS) recognition. In this paper we evaluate the usefulness of these proposed techniques on broadcast news (BN), a similar challenging task. We also perform a set of recognition measurements to understand how close the achieved automatic speech recognition results are to human performance on this task. On two publicly available BN test sets, DEV04F and RT04, our speech recognition system using LSTM and residual network based acoustic models with a combination of n-gram and neural network language models performs at 6.5% and 5.9% word error rate. By achieving new performance milestones on these test sets, our experiments show that techniques developed on other related tasks, like CTS, can be transferred to achieve similar performance. In contrast, the best measured human recognition performance on these test sets is much lower, at 3.6% and 2.8% respectively, indicating that there is still room for new techniques and improvements in this space, to reach human performance levels.
BibTeX
@inproceedings{thomas2019english,
  author = {Samuel Thomas and Masayuki Suzuki and Yinghui Huang and Gakuto Kurata and Zoltán Tüske and George Saon and Brian Kingsbury and Michael Picheny and Tom Dibert and Alice Kaiser-Schatzlein and Bern Samko},
  title = {{English Broadcast News Speech Recognition by Humans and Machines}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=BnE4wb0A8g9},
}

Loading related content...

Figure