TL;DR
This paper introduces BPE-dropout, a novel technique for dynamically augmenting acoustic units in end-to-end Automatic Speech Recognition (ASR) systems. By introducing non-determinism in subword tokenization, BPE-dropout significantly improves recognition of out-of-vocabulary (OOV) words and reduces the need for extensive hyperparameter tuning, particularly in low-resource scenarios.
What Problem Does This Paper Address?
Speech assistants are becoming increasingly common, but their effectiveness hinges on accurate and personalized speech recognition. A major hurdle is handling "out-of-vocabulary" (OOV) words – words the system hasn't encountered during training. This is especially challenging for low-resource languages where large amounts of training data are scarce. Traditional end-to-end ASR models, while efficient, require substantial data to perform well. This paper tackles the dual challenge of improving OOV recognition and achieving good performance in low-resource settings for end-to-end ASR systems.
Methods
The core contribution is BPE-dropout, a dynamic acoustic unit augmentation method. Standard Byte Pair Encoding (BPE) deterministically segments words into subwords. BPE-dropout, however, introduces randomness by randomly discarding some merging rules during the segmentation process with a certain probability p. This leads to:
- Dynamic Tokenization: The same word can be tokenized in multiple ways during training, exposing the model to a richer set of contexts.
- Regularization: By varying the tokenization, the model is less likely to overfit to specific subword sequences.
- Reduced Hyperparameter Search: It alleviates the need to find the "optimal" subword vocabulary size, as the dropout mechanism helps regularize performance across different sizes.
The authors also compare BPE-dropout with ULM (Unigram Language Model) subword regularization, another technique that samples different segmentations based on probabilities.
Datasets
The experiments were conducted on two low-resource telephone conversation datasets from the IARPA Babel program:
- Babel Turkish: 73.40 hours of training data.
- Babel Georgian: 50.52 hours of training data.
Official development sets were used for evaluation, with specific OOV rates noted for each.
Key Insights and Ideas
- Augmentation is Key for Low-Resource OOV: The paper strongly suggests that dynamic augmentation, specifically BPE-dropout, is a powerful tool for improving OOV recognition in data-scarce environments.
- BPE-Dropout is More Than Just Regularization: While it acts as a regularizer, its primary benefit comes from diversifying the acoustic units the model sees during training, leading to better generalization.
- Vocabulary Size Trade-off Mitigated: BPE-dropout helps balance the trade-off between small vocabularies (better saturation) and large vocabularies (more specific tokens), reducing the pressure to find a single optimal size.
- Short Tokens Matter for OOV: The analysis shows that BPE-dropout increases the frequency of shorter tokens (1-3 characters) in OOV word recognition, which are crucial for identifying unseen words.
Key Results
- Consistent WER Reduction: BPE-dropout provided a steady improvement, achieving at least 6% relative Word Error Rate (WER) reduction compared to non-augmented models.
- Significant F-score Improvement: For personalized (OOV-oriented) tasks, the technique led to a 25% relative F-score improvement.
- Competitive Performance: The monolingual Turkish Conformer model with BPE-dropout achieved a 22.2% Character Error Rate (CER) and 38.9% WER, competitive with the best published multilingual systems.
- No Additional Computational Cost: The augmentation is applied during training and does not incur extra computational cost during inference.
- ULM vs. BPE-Dropout: BPE-dropout was found to be more effective than ULM subword regularization in their experiments.
Limitations and Future Work
- Personalization vs. Explicit Techniques: While BPE-dropout improves OOV recognition, it may not fully replace explicit personalization techniques when quality is paramount over speed.
- Frame Subsampling Impact: The effectiveness of BPE-dropout might diminish with higher frame subsampling factors (e.g., 8 or 16) due to shifts in acoustic unit distribution.
- Efficiency for Smart Assistants: The end-to-end systems, while high-quality, might require further enhancements (e.g., model compression, decoding optimization) for direct deployment on smart assistants.
- Data Augmentation for Robustness: The study suggests that augmenting telephone waveforms with synthetic room impulse responses and target microphone data could further improve robustness.
- Future Directions: The authors propose exploring Morfessor EM+Prune, investigating non-deterministic subword tokenization with high frame subsampling, and scheduling the dropout probability during training.
Summary updated