Reducing errors by increasing the error rate: MLP Acoustic Modeling for Broadcast News Transcription

Nelson Morgan, Daniel P. W. Ellis, Eric Fosler‐Lussier, Adam Janin, Brian KingsburyColumbia 1999 · DOI

TL;DR

This paper explores novel acoustic modeling techniques for broadcast news transcription, focusing on improving accuracy by combining new "modulation spectrogram" features with existing models and investigating the impact of model size and training data. They found that while some novel approaches didn't yield immediate gains, combining different feature sets and increasing model complexity significantly reduced word error rates, with gender-dependent modeling offering further moderate improvements.

What Problem Does This Paper Address?

Automatic speech recognition (ASR) systems, particularly for challenging domains like broadcast news, constantly strive to reduce word error rates (WER). While incremental improvements are common, this paper investigates less-explored, potentially more impactful approaches. The authors aim to improve a hybrid Hidden Markov Model (HMM)/Multi-Layer Perceptron (MLP) acoustic modeling system by:

  1. Introducing new auditory-inspired features ("modulation spectrograms") that capture different aspects of speech than traditional methods like PLP.
  2. Exploring nonlinear segment normalization to better adapt models to individual speaker characteristics.
  3. Systematically studying the interplay between model size (number of MLP parameters) and training data size to understand how to best leverage large datasets.
  4. Investigating the potential of gender-dependent acoustic models to further reduce errors.

The context is the ongoing effort to build robust ASR systems for broadcast news, a domain characterized by diverse speakers, acoustic conditions, and speech styles.

Methods

The researchers developed and tested several experimental systems, primarily using a hybrid HMM/MLP architecture. Key methods include:

  • Modulation Spectrogram (MSG) Features: A novel feature extraction technique inspired by auditory processing. These features are designed to provide explicit automatic gain control and modulation filtering for each critical band energy contour, aiming to be more robust to acoustic variations than traditional PLP or RASTA-PLP features. MSG features are represented in spectral form and consist of two groups of 14 features each, processed with different low-pass and band-pass filters.
  • Feature Combination: MSG features were combined with conventional PLP features. Initially, this was done by multiplying the frame-level likelihoods from separate models. Later, they explored combining subsystems at the hypothesis level.
  • Nonlinear Segment Normalization: Instead of traditional methods like Maximum Likelihood Linear Regression (MLLR) (which is not directly applicable to MLPs), they augmented the framewise MLP input with segment-level statistics. These statistics, such as standard deviation of feature elements, were intended to capture "speaker character" and reduce across-speaker variability.
  • Model Size and Training Data Interaction: Extensive experiments were conducted by training MLPs with varying numbers of hidden layers (500 to 8000 units) and different sizes of training data (from 1/8 to the full 142 hours of broadcast news data).
  • Gender-Based Modeling: Networks were trained separately for male and female speakers, initialized with weights from a gender-independent network. During recognition, the system could choose between gender-specific or gender-independent models based on utterance likelihood. They also explored a simpler frame-level combination of gender-dependent and independent models.

Datasets

The primary dataset used was the Broadcast News speech corpus. Specifically, they utilized 74 hours of data from 1997 for initial experiments and expanded to 142 hours from 1997 and 1998 for later, larger model training. The data was often low-pass filtered at 4 kHz to simulate telephone bandwidth, as used in some of their partner systems.

Key Insights and Ideas

  • Combining Weaknesses Can Create Strength: While MSG features alone sometimes performed worse than PLP, combining them with PLP features significantly reduced word error rates, especially for degraded speech conditions. This highlights the benefit of diverse feature sets.
  • Model Complexity and Data are Key: The paper strongly suggests that for their MLP architecture, increasing both the number of model parameters (hidden units) and the amount of training data consistently leads to significant reductions in word error rate, even with context-independent models.
  • Novel Normalization Challenges: The attempt to use segment-level statistics for nonlinear normalization didn't yield significant improvements over standard mean/variance normalization. This suggests that simpler normalization might already capture much of the desired effect, or that more sophisticated methods are needed.
  • Gender Matters (Moderately): Incorporating gender-dependent acoustic models provided moderate but consistent reductions in word error rate, confirming its utility as an enhancement.
  • Exploration is Valuable: The authors emphasize the importance of exploring less conventional approaches, even if they don't immediately yield the best results, as they can lead to breakthroughs or a better understanding of system limitations.

Key Results

  • Combining PLP and MSG features (multiplying frame likelihoods) reduced word error rates from 27.6% (MLP/MSG) and 25.1% (RNN/PLP) to 23.3% on the first evaluation set and 22.7% on the second.
  • Increasing the MLP hidden layer size from 4000 to 8000 units, trained on the full 142 hours of data, reduced the word error rate of their best system from 29.3% to 26.8%.
  • Nonlinear segment normalization experiments showed only marginal improvement (35.4% vs. 36.0% on a development test set) and were not used in the final evaluation.
  • Gender-based modeling improved the best gender-independent MSG net from 29.7% WER to 27.9%.

Limitations and Future Work

  • Nonlinear Segment Normalization: The authors acknowledge that their attempt at nonlinear segment normalization was not successful and suggest that future work might explore other features or methods to incorporate speaker identity.
  • Context-Dependent Modeling: They note that context-dependent modeling, which their Cambridge partners used, was not explored in their MLP system, but is a known avenue for improvement.
  • Adaptation: Similar to context-dependent modeling, adaptation techniques were not a primary focus but are considered important for future systems.
  • Computational Cost: The paper highlights the immense computational effort required for training large networks, suggesting the need for efficient custom hardware (like their Torrent vector microprocessors) and optimized training procedures.
  • Multi-band Recognition: The authors mention that multi-band recognition is an area of ongoing research that they still don't fully understand, indicating a potential future direction.

Summary updated

Abstract

We describe some aspects of a Broadcast News recognition system based on hybrid HMM/MLP acoustic modeling. These include the use of novel 'modulation spectrogram' features which are combined with conventional models at the posterior probability level, some experiments with nonlinear segment normalization, and an investigation of the interaction of model size and training set size for an multilayer perceptron (MLP) acoustic classifier. We also report preliminary results of incorporating gender-dependence into this system.
BibTeX
@inproceedings{morgan1999reducing,
  author = {Nelson Morgan and Daniel P. W. Ellis and Eric Fosler‐Lussier and Adam Janin and Brian Kingsbury},
  title = {{Reducing errors by increasing the error rate: MLP Acoustic Modeling for Broadcast News Transcription}},
  booktitle = {Columbia},
  year = {1999},
  doi = {10.7916/d8mw2sj1},
}
Figure