TL;DR
This paper explores how the order in which training data is presented, not just the data itself, significantly impacts the performance of end-to-end Automatic Speech Recognition (ASR) systems. They introduce and compare novel "curriculum learning" strategies that go beyond simple duration-based sorting, demonstrating improved convergence and comparable or better accuracy, even with less training data.
What Problem Does This Paper Address?
End-to-end (e2e) ASR models have become increasingly popular due to their unified architecture, but they often require substantial amounts of training data and careful training strategies to achieve performance comparable to traditional hybrid models. A key challenge is how to effectively train these models, especially when data is limited. The authors posit that, much like humans learn progressively, the sequence of training examples matters. They aim to improve e2e ASR training by investigating and proposing new methods for "Curriculum Learning" (CL), a technique that structures training data from easy to hard.
Methods
The core of this paper lies in exploring different "scoring functions" to define the difficulty of training examples for CL. They compare several approaches:
- Metadata-based (Duration-based): The standard approach, sorting utterances by their length (shorter = easier).
- Loss-based (Seq2Seq Scoring - S2S): Uses the model's own loss on an utterance as a difficulty indicator. Higher loss means harder.
- Metric-based (WER Scoring - WS): Uses the Word Error Rate (WER) from a decoding step as a difficulty metric. Higher WER means harder.
- Uniform Mixing (WS-M, S2S-M): Combines the above with a strategy of mixing a small percentage of easy examples with medium and hard ones within an epoch to prevent overfitting.
- Transfer Learning (T-WS-M, T-S2S-M): Leverages an external, pre-trained "teacher" model to score the difficulty of utterances.
- Pacing Function: Instead of reordering the entire dataset, this method controls the proportion of data seen at each epoch, gradually increasing it.
They also introduce an "adaptive" approach where the difficulty scores are updated dynamically during training, contrasting with "static" approaches that fix the order upfront.
Datasets
The experiments were conducted on the Lahjoita puhetta (Donate speech) corpus, a collection of colloquial Finnish speech.
- Training set: Initially 1601.5 hours, segmented into 10-second chunks, resulting in 1382 hours.
- Validation set: Initially 10.5 hours, reduced to 8.3 hours.
- Test set: Initially 10.4 hours, reduced to 8.87 hours.
The paper also presents results for models trained on subsets of this data: 10% (138 hours) and 30% (414 hours).
Key Insights and Ideas
- Order Matters: The way training data is presented is as crucial as the data itself for e2e ASR.
- Beyond Duration: Simple duration-based sorting is not always optimal. Metric-based (WER) and loss-based scores can provide more nuanced difficulty assessments.
- Adaptive is Better: Dynamically updating difficulty scores during training (adaptive CL) offers advantages over static curricula.
- Mixing Prevents Overfitting: Combining easy examples with harder ones within an epoch (uniform mixing) helps mitigate overfitting, especially early in training.
- Pacing for Efficiency: Using a pacing function can significantly reduce training time while maintaining competitive performance.
- Stabilizing Effect: Metric-based CL strategies (like WS-M) showed a stabilizing effect, reducing the variance in performance across multiple training runs, suggesting robustness against random initialization.
Key Results
- The WER Scoring with Uniform Mixing (WS-M) strategy consistently outperformed other methods on both validation and test sets across different data sizes.
- The Random Shuffling (RS) strategy also consistently outperformed the baseline (duration-based) method.
- Metric-based CL strategies (WS-M) demonstrated a stabilizing effect, reducing the standard deviation of results compared to the baseline.
- Using a pacing function with WS-M or T-WS-M allowed models to achieve performance comparable to the baseline while training for only about 60% of the time.
- The S2S-M strategy tended to underfit, while the WS strategy tended to overfit, highlighting the importance of careful curriculum design.
Limitations and Future Work
- Training Time Overhead: While WS-M achieved better accuracy, it incurred a higher training time overhead (15%) compared to the baseline due to the need to re-calculate scores. Transfer learning approaches showed minimal overhead.
- Future Work: The authors plan to develop new scoring and pacing functions that incorporate more information from the neural network itself. They also aim to test their approaches on larger, English datasets to validate their effectiveness at a broader scale.
Summary updated