Acoustic Modelling from the Signal Domain Using CNNs

Pegah Ghahremani, Vimal Manohar, Daniel Povey, Sanjeev KhudanpurInterspeech 2016 · DOI

Introduction

Traditional Automatic Speech Recognition (ASR) systems have long relied on hand-crafted spectral features, such as Mel-Frequency Cepstral Coefficients (MFCCs) or Perceptual Linear Prediction (PLP) coefficients. While these features are designed to mimic human auditory perception, they may inadvertently discard acoustic information critical for minimizing Word Error Rate (WER). This paper investigates the feasibility of learning acoustic models directly from raw audio waveforms—the "signal domain"—using Convolutional Neural Networks (CNNs).

Learning from raw signals presents significant challenges, including high input dimensionality (8–16 kHz sampling rates) and sensitivity to small temporal shifts. The authors address these issues through a specialized architecture featuring a novel "Network-in-Network" (NIN) approach and a custom statistics extraction layer.

Methodology

The proposed architecture is designed to handle the high variance of raw audio while extracting robust features for classification.

Feature Extraction Block

The model replaces the traditional Fourier Transform with a 1D convolutional layer. This layer operates directly on the raw signal with a small step size (1.25ms), effectively learning a bank of filters optimized for the task.

Figure 4: The Feature Extraction Block architecture, illustrating the integration of 1D convolution filters and NIN blocks for signal processing.

Network-in-Network (NIN) Nonlinearity

To aggregate temporal information, the authors utilize "micro neural network blocks" instead of standard max-pooling. These NIN blocks consist of interleaved block-diagonal matrices and ReLU activations. This structure allows the network to learn complex, non-linear compression of temporal data, providing a more flexible alternative to fixed mathematical functions like averaging or maximums.

Statistics Extraction and Data Perturbation

To facilitate speaker adaptation, a statistics extraction layer calculates the mean and variance of hidden unit activations over a moving 2-second window. This captures long-term speaker and channel characteristics. Additionally, to ensure translation invariance, the authors employ data perturbation by randomly shifting raw input samples during training, preventing the model from over-fitting to specific waveform alignments.

Experimental Results

The models were evaluated on the Wall Street Journal (WSJ) and Switchboard datasets.

Performance and Convergence

On the WSJ task (Nov’92 eval), the raw waveform system with NIN achieved a 3.92% WER, a substantial improvement over the 5.28% WER of the MFCC baseline. In the more challenging Switchboard task, the system achieved 16.3% WER, slightly outperforming the MFCC-based equivalent.

Figure 6: Comparison of training log-likelihood across different pooling strategies, demonstrating the superior convergence and performance of NIN pooling.

As shown in Figure 6, the NIN architecture significantly outperforms traditional p-norm pooling and configurations without pooling. Furthermore, the authors observed that the NIN-based model reached optimal performance in fewer iterations than traditional Time Delay Neural Network (TDNN) baselines using MFCCs.

Key Insights and Discussion

The superiority of NIN over traditional pooling suggests that learning how to compress information is more effective than using fixed heuristics. For smaller datasets like WSJ, the inclusion of random sample shifting was found to be critical for generalization.

Interestingly, while iVector adaptation is a standard technique in ASR, it proved less effective for raw waveform models. The authors found that their internal statistics extraction layer provided better speaker-dependent cues, suggesting that the network can autonomously extract relevant speaker information directly from the signal.

Conclusion

This work demonstrates that ASR systems can achieve state-of-the-art results by learning directly from raw audio. By replacing hand-crafted features with a learnable feature extraction block and utilizing NIN for temporal aggregation, the proposed model matches or exceeds the performance of traditional MFCC-based systems while offering faster convergence. Future research will focus on the robustness of these signal-domain models in highly mismatched or noisy acoustic environments.

Summary updated

BibTeX
@inproceedings{ghahremani2016acoustic,
  author = {Pegah Ghahremani and Vimal Manohar and Daniel Povey and Sanjeev Khudanpur},
  title = {{Acoustic Modelling from the Signal Domain Using CNNs}},
  booktitle = {Interspeech},
  year = {2016},
  doi = {10.21437/interspeech.2016-1495},
}
Figure