Acoustic modeling with deep neural networks using raw time signal for LVCSR

Zoltán Tüske, Pavel Golik, Ralf Schlüter, Hermann NeyInterspeech 2014

TL;DR

This paper explores how much speech signal preprocessing is truly necessary for Deep Neural Network (DNN) acoustic models in Large Vocabulary Continuous Speech Recognition (LVCSR). They demonstrate that DNNs can achieve competitive results even when trained on raw audio waveforms, significantly reducing the need for traditional, complex feature extraction pipelines. This opens doors for more flexible and potentially more accurate acoustic modeling by letting the DNN learn features directly from the data.

What Problem Does This Paper Address?

For decades, automatic speech recognition (ASR) systems have relied on carefully engineered acoustic features, like Mel-Frequency Cepstral Coefficients (MFCCs). These features are designed to mimic human auditory perception and are often decorrelated and reduced in dimensionality to be compatible with older modeling techniques like Gaussian Mixture Models (GMMs). While DNNs are more flexible and can handle higher-dimensional, correlated inputs, the question remains: how much of this traditional feature engineering is still beneficial, or can DNNs learn the necessary representations directly from the raw audio signal? This paper investigates this by systematically reducing the preprocessing steps and evaluating the impact on ASR performance.

Methods

The core of this research lies in comparing DNN acoustic models trained on different levels of feature extraction. They start with standard MFCCs and progressively simplify the pipeline, culminating in using the raw audio waveform itself.

  • Feature Extraction Pipelines:
    • MFCCs: Standard features derived from Mel-scale filter banks and DCT.
  • Critical Band Energies (CRBE): Similar to MFCCs but with potentially higher frequency resolution.
  • FFT Magnitude Spectrum: Direct spectral representation from the Short-Time Fourier Transform (STFT).
  • Raw Time Signal: The unprocessed audio samples, with only mean/variance normalization.
  • DNN Architecture: A standard hybrid DNN-HMM system with 6 hidden layers, each containing 2000 units. The output layer has 4500 nodes for generalized triphones.
  • Activation Functions: They compare traditional sigmoid activation functions with Rectified Linear Units (ReLUs), noting ReLUs' potential for improving performance on challenging tasks.
  • Weight Initialization: To address the challenge of learning from raw signals, they experiment with initializing the first hidden layer weights with impulse responses of audiological filters (Gammatone filters).
  • Feature Combination: They also explore combining different feature types (MFCC, PLP, Gammatone) to see if the DNN can leverage complementary information.

Datasets

  • Training: 50 hours of speech from the Quaero [17] English database (train11).
  • Development/Evaluation: Approximately 3.5 hours of speech each.
  • Large-Scale Training: Some experiments were conducted on a larger 250-hour dataset from the same corpus.

Key Insights and Ideas

  • DNNs Learn Features: The most striking finding is that DNNs can learn meaningful acoustic features directly from the raw time signal, significantly reducing the need for complex, hand-crafted feature extraction.
  • Activation Function Matters: Switching from sigmoid to Rectified Linear Units (ReLUs) dramatically closes the performance gap between traditional features (MFCC) and the raw time signal, especially for systems with higher error rates.
  • Raw Signal Potential: While raw time signal initially performed worse, with ReLUs and sufficient training data, it becomes a strong contender, even outperforming MFCCs in some configurations.
  • Learned Filters Mimic Auditory Perception: Analysis of the first layer weights of DNNs trained on raw signals reveals that the network learns band-pass filters that resemble audiological filter banks, distributed across the frequency spectrum in a non-linear fashion.
  • Feature Combination is Powerful: Combining different feature types (MFCC, PLP, Gammatone) allows the DNN to learn from the subtle differences between them, leading to improved performance.
  • Manual Initialization Limitations: While initializing the first layer with audiological filters was explored, it didn't significantly improve performance over random initialization, suggesting the DNN can learn these patterns effectively on its own.

Key Results

  • Raw Signal vs. MFCC: Initially, MFCCs with standard preprocessing yielded better results than raw time signals. However, with ReLUs and increased training data, the raw time signal performance significantly improved. For instance, on the evaluation set, a raw time signal DNN with ReLUs achieved 28.5% WER (50h training), compared to 25.3% for an MFCC DNN. With 250h training, this gap narrowed further to 23.5% for raw signal vs. 21.1% for MFCC.
  • Feature Combination: Combining MFCC, PLP, and Gammatone features resulted in the best performance among the tested low-dimensional features, achieving 18.4% WER on the development set and 24.2% on the evaluation set (50h training).
  • Rectified Linear Units (ReLUs): ReLUs consistently improved performance across all feature types compared to sigmoid activation functions, particularly for the raw time signal.
  • Large Scale Training: Training on 250 hours of data further reduced the Word Error Rate (WER) for all feature types, bringing the raw time signal DNN closer to the performance of MFCC-based systems.

Limitations and Future Work

  • Manual Initialization: The authors found that manual initialization of the first layer with audiological filters did not significantly improve performance and even fixing these weights hurt performance. This suggests that the DNN's ability to learn from data is robust, but also that the specific manual initialization might not have been optimal or that the DNN can discover better representations.
  • Feature Dimensionality: While raw time signal offers a reduction in preprocessing, it leads to very high-dimensional input spaces, increasing the number of trainable parameters. The paper mentions techniques like narrow matrix factorization as a countermeasure, but this is an ongoing challenge.
  • Future Work: The paper suggests that presenting the DNN with data at the sampling frequency level allows it to learn non-stationary patterns localized in time, which is a significant advantage over frame-based processing. Further exploration of this capability, especially with larger datasets and more advanced DNN architectures, is a promising direction. The success of feature combination also hints at the potential for more sophisticated ways to integrate diverse information sources.

Summary updated

BibTeX
@inproceedings{tuske2014acoustic,
  author = {Zoltán Tüske and Pavel Golik and Ralf Schlüter and Hermann Ney},
  title = {{Acoustic modeling with deep neural networks using raw time signal for LVCSR}},
  booktitle = {Interspeech},
  year = {2014},
}
Figure