A lecture transcription system combining neural network acoustic and language models

Peter Bell, Hitoshi Yamamoto, Pawel Swietojanski, Youzheng Wu, Fergus McInnes, Chiori Hori, Steve RenalsInterspeech 2013 · OpenReview

Abstract

This paper describes a high-performance automatic speech recognition (ASR) system specifically designed for the transcription of TED lectures. Developed for the International Workshop on Spoken Language Translation (IWSLT) evaluation, the system leverages a "best-of-both-worlds" approach by combining advanced acoustic models from the University of Edinburgh with sophisticated language models from NICT Japan. By integrating Multi-level Adaptive Networks (MLAN) and factored Recurrent Neural Networks (fRNN), the authors achieved a significant 16% relative reduction in Word Error Rate (WER) over existing benchmarks.

Introduction

Transcribing lectures like TED talks presents unique challenges for ASR. While audio quality is typically high due to head-mounted microphones, the vocabulary is vast and the topics are highly diverse, ranging from technical scientific subjects to social commentary. Traditional systems based on Hidden Markov Models and Gaussian Mixture Models (HMM-GMM) have historically struggled to scale with this complexity. This research aims to break through performance plateaus by leveraging Deep Neural Networks (DNNs) and effectively utilizing out-of-domain data to improve in-domain performance.

Methodology

Multi-level Adaptive Networks (MLAN)

The core acoustic innovation is the Multi-level Adaptive Network (MLAN). Rather than training exclusively on TED data, the system first trains a DNN on the AMI corpus—a dataset of multiparty meeting recordings. The outputs of this "out-of-domain" (OOD) network serve as additional features for a second, "in-domain" network trained on TED talks. This hierarchical approach allows the system to learn robust phonetic patterns from a broad dataset without being negatively impacted by the acoustic differences between meetings and lectures.

The MLAN architecture utilizes out-of-domain posterior features to enhance in-domain training.

Acoustic Configurations

The system utilizes DNNs in two distinct configurations:

  1. Tandem: DNNs act as feature extractors for a GMM.
  2. Hybrid: DNNs directly estimate state probabilities.

The final output is generated by combining these approaches via ROVER (Recognizer Output Voting Error Reduction), which maximizes the strengths of both modeling techniques.

Factored Recurrent Neural Network Language Models (fRNNLM)

Standard language models (LMs) typically analyze sequences of words. The authors implemented factored RNNs, which incorporate additional "factors" such as Part-of-Speech (POS) tags. By understanding the grammatical structure alongside the surface form of words, the fRNNLM significantly improves the prediction of rare words and maintains coherence across the diverse topics found in TED talks.

System Architecture and Decoding

The transcription process follows a multi-pass decoding pipeline. It begins with initial decoding using OOD tandem features and a 3-gram LM. This is followed by speaker adaptation using Constrained Maximum Likelihood Linear Regression (CMLLR) to tune the model to individual lecturers. Subsequent passes involve lattice rescoring with a 4-gram LM and 100-best list rescoring with the fRNN LM before the final ROVER combination.

The multi-pass decoding pipeline integrates speaker adaptation and advanced language model rescoring.

Experimental Setup

  • Acoustic Data: 153 hours of segmented TED talks (in-domain) and 127 hours of the AMI meeting corpus (out-of-domain).
  • Language Data: 2.4 million tokens of TED transcripts, supplemented by 4.5 million tokens of News Commentary and 2.7 billion tokens from the English Gigaword 5th edition.
  • Evaluation: Benchmarked on IWSLT dev2010, tst2010, tst2011, and tst2012 sets.

Key Results

The integrated system demonstrated a major leap in transcription accuracy:

  • WER Reduction: On the tst2011 benchmark, the system achieved a 9.3% WER, a 16% relative improvement over the previous year's best system.
  • LM Impact: Transitioning from a standard 3-gram LM to the factored RNNLM provided a consistent absolute reduction in WER of 1.5% to 2.0%.
  • Adaptation: Despite the power of DNNs, unsupervised speaker adaptation (CMLLR) remained a critical component for achieving competitive error rates.

Conclusion

This research demonstrates that cross-domain synergy—using mismatched data like office meetings to bolster lecture transcription—is highly effective when implemented via MLAN. While the computational cost of training fRNNs on massive datasets remains high, the resulting gains in accuracy set a new standard for the field. Future work will focus on more direct adaptation methods for hybrid DNN models and further exploration of bottleneck feature dimensions.

Summary updated

BibTeX
@inproceedings{bell2013lecture,
  author = {Peter Bell and Hitoshi Yamamoto and Pawel Swietojanski and Youzheng Wu and Fergus McInnes and Chiori Hori and Steve Renals},
  title = {{A lecture transcription system combining neural network acoustic and language models}},
  booktitle = {Interspeech},
  year = {2013},
  url = {https://openreview.net/forum?id=NNTgTOCTMal},
}
Figure