Small energy masking for improved neural network training for end-to-end speech recognition

Chanwoo Kim, Kwangyoun Kim, Sathish Reddy Indurthi

Abstract

In this paper, we present a Small Energy Masking (SEM) algorithm, which masks inputs having values below a certain threshold. More specifically, a time-frequency bin is masked if the filterbank energy in this bin is less than a certain energy threshold. A uniform distribution is employed to randomly generate the ratio of this energy threshold to the peak filterbank energy of each utterance in decibels. The unmasked feature elements are scaled so that the total sum of the feature values remain the same through this masking procedure. This very simple algorithm shows relatively 11.2 % and 13.5 % Word Error Rate (WER) improvements on the standard LibriSpeech test-clean and test-other sets over the baseline end-to-end speech recognition system. Additionally, compared to the input dropout algorithm, SEM algorithm shows relatively 7.7 % and 11.6 % improvements on the same LibriSpeech test-clean and test-other sets. With a modified shallow-fusion technique with a Transformer LM, we obtained a 2.62 % WER on the LibriSpeech test-clean set and a 7.87 % WER on the LibriSpeech test-other set.

Introduction

Automatic Speech Recognition (ASR) has seen significant advancements with deep learning, yet overfitting remains a major challenge. Standard regularization techniques like Dropout randomly deactivate neurons to improve generalization. However, in speech signals, features are not equally informative or robust. Low-energy time-frequency bins are particularly susceptible to additive noise. This paper proposes Small Energy Masking (SEM), which selectively masks these low-energy components to force models to rely on more robust, high-energy features.

The Small Energy Masking (SEM) Algorithm

The SEM algorithm operates on the filterbank energy of the speech signal. Unlike standard dropout, which is applied element-wise with a fixed probability, SEM uses the physical properties of the signal to determine masking.

Block diagram of the Small Energy Masking (SEM) algorithm.

The process involves five key steps:

  1. Energy Calculation: Compute the filterbank energy for each time-frequency bin.
  2. Peak Detection: Identify the peak energy ($e_{peak}$) of the utterance, typically using the 95th percentile to avoid outliers.
  3. Random Thresholding: A threshold ratio ($\eta_{th}$) is randomly sampled from a uniform distribution (e.g., -80 dB to 0 dB relative to the peak).
  4. Binary Masking: Bins with energy below the threshold $e_{th} = e_{peak} + \eta_{th}$ are zeroed out.
  5. Energy Preservation: Unmasked features are scaled up to maintain the total energy sum, ensuring the network isn't biased by changes in input magnitude.

Visualizing the Masking Effect

SEM acts as a form of "informed sparsity." By targeting low-energy bins, it mimics the effects of noise corruption, serving as a specialized data augmentation technique.

Spectrograms illustrating the effect of different energy masking thresholds.

The randomization of the threshold is crucial. Fixed thresholds can degrade performance, whereas random thresholds force the model to be robust across varying levels of signal sparsity.

Experimental Results

The SEM algorithm was evaluated on the LibriSpeech corpus using an end-to-end architecture consisting of a 6-layer Bi-LSTM encoder and a 1-layer LSTM decoder.

  • Performance Gains: SEM achieved an 11.2% relative improvement in Word Error Rate (WER) on the test-clean set and a 13.5% improvement on test-other compared to the baseline.
  • Comparison to Dropout: SEM outperformed standard input dropout by 7.7% on clean speech and 11.6% on noisy speech.
  • State-of-the-Art Integration: When combined with a Transformer-based Language Model using shallow fusion, the system reached a WER of 2.62% on test-clean and 7.87% on test-other.

Conclusion

Small Energy Masking provides a simple yet effective way to regularize speech recognition models by leveraging the inherent structure of speech signals. While highly effective, future work could explore moving-average peak detection for real-time streaming applications and investigate the additive benefits of combining SEM with other techniques like SpecAugment.

Summary updated

Abstract

In this paper, we present a Small Energy Masking (SEM) algorithm, which masks inputs having values below a certain threshold. More specifically, a time-frequency bin is masked if the filterbank energy in this bin is less than a certain energy threshold. A uniform distribution is employed to randomly generate the ratio of this energy threshold to the peak filterbank energy of each utterance in decibels. The unmasked feature elements are scaled so that the total sum of the feature values remain the same through this masking procedure. This very simple algorithm shows relatively 11.2 % and 13.5 % Word Error Rate (WER) improvements on the standard LibriSpeech test-clean and test-other sets over the baseline end-to-end speech recognition system. Additionally, compared to the input dropout algorithm, SEM algorithm shows relatively 7.7 % and 11.6 % improvements on the same LibriSpeech test-clean and test-other sets. With a modified shallow-fusion technique with a Transformer LM, we obtained a 2.62 % WER on the LibriSpeech test-clean set and a 7.87 % WER on the LibriSpeech test-other set.
BibTeX
@inproceedings{kim2020small,
  author = {Chanwoo Kim and Kwangyoun Kim and Sathish Reddy Indurthi},
  title = {{Small energy masking for improved neural network training for end-to-end speech recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=V3hzQUd_zw3},
}
Figure