Dynamic Acoustic Unit Augmentation with BPE-Dropout for Low-Resource End-to-End Speech Recognition

Aleksandr Laptev, Andrei Andrusenko, Ivan Podluzhny, Anton Mitrofanov, Ivan Medennikov, Yuri Matveev

TL;DR

This paper introduces BPE-dropout, a novel technique for dynamically augmenting acoustic units in end-to-end Automatic Speech Recognition (ASR) systems. By introducing non-determinism in subword tokenization, BPE-dropout significantly improves recognition of out-of-vocabulary (OOV) words and reduces the need for extensive hyperparameter tuning, particularly in low-resource scenarios.

What Problem Does This Paper Address?

Speech assistants are becoming increasingly common, but their effectiveness hinges on accurate and personalized speech recognition. A major hurdle is handling "out-of-vocabulary" (OOV) words – words the system hasn't encountered during training. This is especially challenging for low-resource languages where large amounts of training data are scarce. Traditional end-to-end ASR models, while efficient, require substantial data to perform well. This paper tackles the dual challenge of improving OOV recognition and achieving good performance in low-resource settings for end-to-end ASR systems.

Methods

The core contribution is BPE-dropout, a dynamic acoustic unit augmentation method. Standard Byte Pair Encoding (BPE) deterministically segments words into subwords. BPE-dropout, however, introduces randomness by randomly discarding some merging rules during the segmentation process with a certain probability p. This leads to:

  • Dynamic Tokenization: The same word can be tokenized in multiple ways during training, exposing the model to a richer set of contexts.
  • Regularization: By varying the tokenization, the model is less likely to overfit to specific subword sequences.
  • Reduced Hyperparameter Search: It alleviates the need to find the "optimal" subword vocabulary size, as the dropout mechanism helps regularize performance across different sizes.

The authors also compare BPE-dropout with ULM (Unigram Language Model) subword regularization, another technique that samples different segmentations based on probabilities.

Datasets

The experiments were conducted on two low-resource telephone conversation datasets from the IARPA Babel program:

  • Babel Turkish: 73.40 hours of training data.
  • Babel Georgian: 50.52 hours of training data.

Official development sets were used for evaluation, with specific OOV rates noted for each.

Key Insights and Ideas

  • Augmentation is Key for Low-Resource OOV: The paper strongly suggests that dynamic augmentation, specifically BPE-dropout, is a powerful tool for improving OOV recognition in data-scarce environments.
  • BPE-Dropout is More Than Just Regularization: While it acts as a regularizer, its primary benefit comes from diversifying the acoustic units the model sees during training, leading to better generalization.
  • Vocabulary Size Trade-off Mitigated: BPE-dropout helps balance the trade-off between small vocabularies (better saturation) and large vocabularies (more specific tokens), reducing the pressure to find a single optimal size.
  • Short Tokens Matter for OOV: The analysis shows that BPE-dropout increases the frequency of shorter tokens (1-3 characters) in OOV word recognition, which are crucial for identifying unseen words.

Key Results

  • Consistent WER Reduction: BPE-dropout provided a steady improvement, achieving at least 6% relative Word Error Rate (WER) reduction compared to non-augmented models.
  • Significant F-score Improvement: For personalized (OOV-oriented) tasks, the technique led to a 25% relative F-score improvement.
  • Competitive Performance: The monolingual Turkish Conformer model with BPE-dropout achieved a 22.2% Character Error Rate (CER) and 38.9% WER, competitive with the best published multilingual systems.
  • No Additional Computational Cost: The augmentation is applied during training and does not incur extra computational cost during inference.
  • ULM vs. BPE-Dropout: BPE-dropout was found to be more effective than ULM subword regularization in their experiments.

Limitations and Future Work

  • Personalization vs. Explicit Techniques: While BPE-dropout improves OOV recognition, it may not fully replace explicit personalization techniques when quality is paramount over speed.
  • Frame Subsampling Impact: The effectiveness of BPE-dropout might diminish with higher frame subsampling factors (e.g., 8 or 16) due to shifts in acoustic unit distribution.
  • Efficiency for Smart Assistants: The end-to-end systems, while high-quality, might require further enhancements (e.g., model compression, decoding optimization) for direct deployment on smart assistants.
  • Data Augmentation for Robustness: The study suggests that augmenting telephone waveforms with synthetic room impulse responses and target microphone data could further improve robustness.
  • Future Directions: The authors propose exploring Morfessor EM+Prune, investigating non-deterministic subword tokenization with high frame subsampling, and scheduling the dropout probability during training.

Summary updated

Abstract

With the rapid development of speech assistants, adapting server-intended automatic speech recognition (ASR) solutions to a direct device has become crucial. For on-device speech recognition tasks, researchers and industry prefer end-to-end ASR systems as they can be made resource-efficient while maintaining a higher quality compared to hybrid systems. However, building end-to-end models requires a significant amount of speech data. Personalization, which is mainly handling out-of-vocabulary (OOV) words, is another challenging task associated with speech assistants. In this work, we consider building an effective end-to-end ASR system in low-resource setups with a high OOV rate, embodied in Babel Turkish and Babel Georgian tasks. We propose a method of dynamic acoustic unit augmentation based on the Byte Pair Encoding with dropout (BPE-dropout) technique. The method non-deterministically tokenizes utterances to extend the token's contexts and to regularize their distribution for the model's recognition of unseen words. It also reduces the need for optimal subword vocabulary size search. The technique provides a steady improvement in regular and personalized (OOV-oriented) speech recognition tasks (at least 6% relative word error rate (WER) and 25% relative F-score) at no additional computational cost. Owing to the BPE-dropout use, our monolingual Turkish Conformer has achieved a competitive result with 22.2% character error rate (CER) and 38.9% WER, which is close to the best published multilingual system.
BibTeX
@inproceedings{laptev2021dynamic,
  author = {Aleksandr Laptev and Andrei Andrusenko and Ivan Podluzhny and Anton Mitrofanov and Ivan Medennikov and Yuri Matveev},
  title = {{Dynamic Acoustic Unit Augmentation with BPE-Dropout for Low-Resource End-to-End Speech Recognition}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s21093063},
}
Figure