Small Energy Masking for Improved Neural Network Training for End-To-End Speech Recognition

Chanwoo Kim, Kwangyoun Kim, Sathish Reddy Indurthi

In the world of Automatic Speech Recognition (ASR), bigger models usually mean better performance, but they also bring the persistent headache of overfitting. While standard techniques like Dropout randomly "kill" neurons to force robustness, a new paper from Samsung Research suggests that when it comes to speech data, we should be more selective about what we hide.

TL;DR

The researchers introduce Small Energy Masking (SEM), a data augmentation technique that selectively masks time-frequency bins with low energy during training. By focusing the model on high-energy (and typically more robust) features while randomizing the masking threshold, they achieved up to a 13.5% relative improvement in Word Error Rate (WER) on the LibriSpeech dataset.

What Problem Does This Paper Address?

Deep learning models for speech recognition are prone to overfitting, especially when trained on limited or noisy data. The standard solution, Dropout, treats all input features equally, randomly zeroing out elements regardless of their content.

However, speech signals aren't uniform. Previous research has shown that low-energy time-frequency bins (the "quiet" parts of a spectrogram) are significantly more susceptible to additive noise. The authors argue that instead of random dropout, we should intentionally mask these "weak" features during training to force the neural network to learn more reliable patterns from the high-energy components of the speech signal.

Methods

The authors propose the Small Energy Masking (SEM) algorithm. Unlike standard dropout, which is applied element-wise with a fixed probability, SEM operates based on the physical properties of the signal:

  1. Energy Calculation: For every utterance, the system calculates the filterbank energy for each time-frequency bin.
  2. Peak Detection: It identifies the "peak energy" ($e_{peak}$) of the utterance (specifically the 95th percentile to avoid outliers).
  3. Random Thresholding: Instead of a fixed cutoff, a threshold ratio ($\eta_{th}$) is randomly sampled from a uniform distribution (typically between -80 dB and 0 dB relative to the peak).
  4. Binary Masking: Any bin with energy lower than this threshold is zeroed out.
  5. Energy Preservation Scaling: To ensure the total sum of feature values remains consistent (preventing the network from being confused by a sudden drop in input magnitude), the unmasked features are scaled up. This is analogous to how units are scaled in standard Dropout.

The system was tested on an end-to-end architecture consisting of a 6-layer Bi-LSTM encoder and a 1-layer LSTM decoder with an attention mechanism.

Datasets

The researchers used the standard LibriSpeech corpus:

  • Training: 960 hours of read English speech.
  • Evaluation: The official test-clean (high-quality audio) and test-other (more challenging audio) sets.

Key Insights and Ideas

  • Informed Sparsity: SEM is essentially "informed dropout." By targeting low-energy bins, it mimics the way noise corrupts speech, effectively acting as a specialized form of data augmentation.
  • The Power of Randomization: One of the most critical findings (shown in Table 3 of the paper) is that a fixed masking threshold actually hurts performance. The magic lies in the randomization of the threshold for every utterance, which prevents the model from simply learning to ignore a specific energy floor and instead forces it to be robust across various levels of signal sparsity.
  • Complementary to Language Models: The technique works well in tandem with modern decoding strategies. When combined with "shallow fusion" (using a Transformer-based Language Model during decoding), the error rates dropped significantly.

Key Results

The SEM algorithm showed impressive gains over both the baseline and standard input dropout:

  • Vs. Baseline: SEM improved WER by 11.2% on test-clean (dropping from 4.19% to 3.72%) and 13.5% on test-other (13.47% to 11.65%).
  • Vs. Input Dropout: SEM outperformed standard dropout by 7.7% and 11.6% on the clean and noisy sets, respectively.
  • State-of-the-Art Results: By combining SEM with a Transformer Language Model, the researchers achieved a WER of 2.62% on test-clean and 7.87% on test-other.

Limitations and Future Work

While the paper demonstrates clear success, a few areas remain for further exploration:

  • Hyperparameter Sensitivity: The performance depends heavily on the range of the uniform distribution ($\eta_a, \eta_b$). Finding the "sweet spot" (-80 dB to 0 dB in this case) requires empirical tuning.
  • Interaction with SpecAugment: The paper mentions SEM is different from SpecAugment (which masks random blocks of frequency/time), but it does not provide a head-to-head comparison or a study on whether using both techniques simultaneously yields additive benefits.
  • Streaming Constraints: The authors note that their "global" peak energy calculation isn't perfectly suited for real-time streaming (where you don't have the whole utterance yet), suggesting that a moving-average peak detection might be needed for production environments.

Summary updated

Abstract

In this paper, we present a Small Energy Masking (SEM) algorithm, which masks inputs having values below a certain threshold. More specifically, a time-frequency bin is masked if the filterbank energy in this bin is less than a certain energy threshold. A uniform distribution is employed to randomly generate the ratio of this energy threshold to the peak filterbank energy of each utterance in decibels. The unmasked feature elements are scaled so that the total sum of the feature values remain the same through this masking procedure. This very simple algorithm shows relatively 11.2% and 13.5% Word Error Rate (WER) improvements on the standard LibriSpeech test-clean and test-other sets over the baseline end-to-end speech recognition system. Additionally, compared to the input dropout algorithm, SEM algorithm shows relatively 7.7% and 11.6% improvements on the same LibriSpeech test-clean and test-other sets. With a modified shallow-fusion technique with a Transformer LM, we obtained a 2.62% WER on the Lib-riSpeech test-clean set and a 7.87% WER on the LibriSpeech test-other set.
BibTeX
@inproceedings{kim2020small,
  author = {Chanwoo Kim and Kwangyoun Kim and Sathish Reddy Indurthi},
  title = {{Small Energy Masking for Improved Neural Network Training for End-To-End Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=V3hzQUd_zw3},
}

Loading related content...

Figure