Advances in lecture recognition: the ISL RT-06s evaluation system

Christian Fügen, Matthias Wölfel, John W. McDonough, Shajith Ikbal, Florian Kraft, Kornel Laskowski, Mari Ostendorf, Sebastian Stüker, Ken'ichi KumataniInterspeech 2006 · OpenReview

Abstract

This paper describes the 2006 lecture recognition system developed at the Interactive Systems Laboratories (ISL), for individual head-microphone (IHM), single distant microphone (SDM), and multiple distant microphones (MDM) conditions. It was evaluated in the RT-06S rich transcription meeting evaluation sponsored by the US National Institute of Standards and Technologies (NIST). We describe the principal differences between our current system and those submitted in previous years, namely, improved acoustic and language models, cross-adaptation between systems with different front-ends and phoneme sets, and the use of various automatic speech segmentation algorithms. Our system achieved word error rates of 38.5% (53.4%) and 22.9% (32.2%), respectively, on the MDM and IHM conditions of the RT-05S (RT-06S) lecture evaluation set.

Introduction

Transcribing academic lectures presents a unique set of challenges for automatic speech recognition (ASR) systems. Unlike controlled studio environments, lectures often involve distant microphones that capture significant room reverberation, ambient noise, and audience cross-talk. The ISL RT-06S system was specifically designed to address these hurdles within the context of the NIST Rich Transcription 2006 evaluation, focusing on three primary scenarios: Individual Head-Microphones (IHM), Single Distant Microphones (SDM), and Multiple Distant Microphones (MDM).

Methodology

The ISL system utilizes an ensemble-style architecture that prioritizes diversity in acoustic modeling and efficiency in signal processing. Key technical innovations include:

  • Dual Front-Ends and Cross-Adaptation: The system employs two distinct acoustic front-ends: standard FFT-based MFCCs and Minimum Variance Distortionless Response (MVDR) spectral envelopes. MVDR is particularly effective at resolving low-frequency regions and ignoring noise in low-energy segments. The "secret sauce" of the system is cross-adaptation, where hypotheses generated by the FFT-based system are used to adapt the MVDR-based system, and vice versa.
  • Blind Channel Combination (BCC): For the multi-microphone (MDM) task, the system avoids the computational cost of decoding every channel. Instead, it selects channels with the highest Signal-to-Noise Ratio (SNR) to create a single optimized waveform. This approach reduced computational overhead by 70% while improving SNR by 2 dB.
  • Advanced Segmentation: A multi-stage speech activity detector using a Multi-Layer Perceptron (MLP) was implemented to distinguish speech from non-speech, specifically targeting the removal of cross-talk in the IHM condition.

Data and Training

To ensure robustness, the models were trained on a diverse corpus of speech data totaling hundreds of hours. This included meeting data from CMU, ICSI, and NIST, as well as lecture-specific data from the TED and CHIL databases. For language modeling, the team augmented standard transcripts with approximately 150 million words scraped from the web, filtered by perplexity to match the lecture domain.

Table 1: Distribution of training data hours across various corpora used for acoustic modeling.

Key Results and Insights

The system demonstrated that acoustic diversity is a primary driver of performance. Cross-adapting systems with different front-ends and phoneme sets (Standard vs. PRONLEX) yielded a ~1.5% absolute improvement in Word Error Rate (WER).

Table 2: Word Error Rate (WER) results for various experimental configurations and decoding passes.

In the final evaluation, the system achieved the following:

  • IHM (Close-talk): A final WER of 22.9% on the development set and 32.2% on the evaluation set.
  • MDM (Multiple Distant Mics): A WER of 38.5% on the development set and 53.4% on the evaluation set.
  • Efficiency: The BCC approach provided a 4% relative WER improvement over the single microphone condition while remaining significantly faster than traditional multi-channel decoding methods.

Discussion and Future Work

While the ISL RT-06S system represents a significant advance in lecture transcription, challenges remain. The system currently operates at 110x to 190x real-time, limiting its use to offline research. Furthermore, overlapping speech—where multiple participants speak simultaneously—remains a major source of error, particularly in the RT-06S evaluation set. Future research will focus on improving real-time factors and developing more sophisticated methods for handling phonetic dictionary integration and speech overlap.

Summary updated

Abstract

This paper describes the 2006 lecture recognition system developed at the Interactive Systems Laboratories (ISL), for individual headmicrophone (IHM), single distant microphone (SDM), and multiple distant microphones (MDM) conditions. It was evaluated in RT-06S rich transcription meeting evaluation sponsored by the US National Institute of Standards and Technologies (NIST). We describe the principal differences between our current system and those submitted in previous years, namely, improved acoustic and language models, cross adaptation between systems with different front-ends and phoneme sets, and the use of various automatic speech segmentation algorithms. Our system achieved word error rates of 38.5% (53.4%) and 22.9% (32.2%), respectively, on the MDM and IHM conditions of the RT-05S (RT-06S) lecture evaluation set.
BibTeX
@inproceedings{fugen2006advances,
  author = {Christian Fügen and Matthias Wölfel and John W. McDonough and Shajith Ikbal and Florian Kraft and Kornel Laskowski and Mari Ostendorf and Sebastian Stüker and Ken'ichi Kumatani},
  title = {{Advances in lecture recognition: the ISL RT-06s evaluation system}},
  booktitle = {Interspeech},
  year = {2006},
  url = {https://openreview.net/forum?id=TSNtWBVZ1q},
}
Figure