Feature Learning in Deep Neural Networks - A Study on Speech Recognition Tasks

Dong Yu, Michael L. Seltzer, Jinyu Li, Jui-Ting Huang, Frank Seide

TL;DR

This paper investigates why Deep Neural Networks (DNNs) outperform traditional models in speech recognition. The authors demonstrate that DNNs excel by learning internal representations that are increasingly invariant to variations in speech, such as speaker differences or noise, as the network depth increases. This inherent robustness allows DNNs to achieve state-of-the-art performance without requiring explicit adaptation techniques.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) has been a long-standing challenge due to the inherent variability in human speech. Factors like different accents, speaking rates, emotional states, background noise, and microphone quality all contribute to significant differences between training data and real-world test conditions. Traditional ASR systems, often based on Gaussian Mixture Models (GMMs) within Hidden Markov Models (HMMs), struggle with this variability, leading to performance degradation. While various adaptation techniques exist to mitigate these mismatches, they often add complexity and latency. This paper aims to understand how DNNs, which have recently shown superior performance, achieve this robustness and whether they can eliminate the need for these complex adaptation methods.

Methods

The authors interpret DNNs as a two-step process: first, a series of non-linear transformations that learn feature representations, and second, a log-linear classifier that uses these representations. They analyze how the internal representations change with network depth. Specifically, they investigate how small perturbations in the input affect the activations in higher layers, hypothesizing that deeper layers become more invariant. They also examine the trade-off between network depth and width, and the impact of training data representativeness on generalization. The core of their analysis involves empirical studies on speech recognition tasks, measuring performance metrics like Word Error Rate (WER) and analyzing internal layer activations.

Datasets

  • Switchboard (SWB) Hub5'00: Used for evaluating the impact of network depth and speaker adaptation.
  • Aurora 4: A benchmark for evaluating robustness to environmental distortions (noise and channel mismatch). This dataset is based on the Wall Street Journal (WSJO) corpus and includes multi-condition training and test sets with various noises and SNRs.
  • Mobile Voice Search (VS) corpus: Used for experiments on mixed-bandwidth speech recognition.

Key Insights and Ideas

  • Feature Learning as Invariance Extraction: The primary insight is that DNNs learn features that are progressively more invariant to variations in the input speech signal as the network depth increases. This is a key reason for their improved performance over shallow models.
  • Deeper is Better (Up to a Point): Increasing the depth of the DNN consistently leads to lower Word Error Rates, indicating enhanced discriminative capabilities. However, the gains diminish with extreme depth, suggesting a saturation point.
  • The Importance of Representative Training Data: DNNs are powerful but cannot "learn from nothing." They require training data that sufficiently covers the expected variations in the test data. If test samples deviate significantly from the training distribution, DNN performance degrades.
  • Inherent Robustness Eliminates Need for Explicit Adaptation: The learned invariant features mean that DNNs can achieve performance comparable to or better than GMM-based systems that rely on explicit adaptation techniques for speaker, bandwidth, or environmental variations.

Key Results

  • Improved WER with Depth: On the Switchboard task, increasing DNN depth from 1 to 9 hidden layers (with 2k units per layer) reduced WER from 24.2% to 17.0%.
  • Mixed-Bandwidth Recognition: A DNN trained on mixed-bandwidth data (both 8kHz and 16kHz) achieved significantly better performance on both narrowband and wideband test sets compared to a DNN trained only on wideband data. This highlights the importance of diverse training data.
  • Speaker Invariance: Deep DNNs showed a much smaller gain from speaker adaptation techniques (VTLN, fMLLR/fDLR) compared to GMM-HMMs and shallow networks, indicating that they already learn speaker-invariant features.
  • Environmental Robustness: On the Aurora 4 task, a DNN achieved competitive results, matching or exceeding systems that required complex adaptation procedures for noise and channel distortion, without needing multiple recognition passes or model adaptation.

Limitations and Future Work

  • Extrapolation Limits: The paper explicitly states that DNNs struggle to generalize to test samples that are substantially different from the training data. This means that if the training data is not representative of all possible variations, DNNs will not perform well.
  • Trade-off in Network Design: While deeper networks are beneficial, there's a trade-off between layer width, performance gains, and the increased cost of training and decoding.
  • Future Work: The authors suggest that further research could explore how to best design training strategies to ensure DNNs see representative samples for various conditions, thereby maximizing their generalization capabilities. The paper also implicitly points towards the continued development of DNN architectures for ASR.

Summary updated

Abstract

Recent studies have shown that deep neural networks (DNNs) perform significantly better than shallow networks and Gaussian mixture models (GMMs) on large vocabulary speech recognition tasks. In this paper, we argue that the improved accuracy achieved by the DNNs is the result of their ability to extract discriminative internal representations that are robust to the many sources of variability in speech signals. We show that these representations become increasingly insensitive to small perturbations in the input with increasing network depth, which leads to better speech recognition performance with deeper networks. We also show that DNNs cannot extrapolate to test samples that are substantially different from the training examples. If the training data are sufficiently representative, however, internal features learned by the DNN are relatively stable with respect to speaker differences, bandwidth differences, and environment distortion. This enables DNN-based recognizers to perform as well or better than state-of-the-art systems based on GMMs or shallow networks without the need for explicit model adaptation or feature normalization.
BibTeX
@inproceedings{yu2013feature,
  author = {Dong Yu and Michael L. Seltzer and Jinyu Li and Jui-Ting Huang and Frank Seide},
  title = {{Feature Learning in Deep Neural Networks - A Study on Speech Recognition Tasks}},
  booktitle = {International Conference on Learning Representations},
  year = {2013},
  url = {https://openreview.net/forum?id=kk_XkMO0-dP8W},
}
Figure