TL;DR
This paper introduces two novel semi-supervised learning (SSL) methods to improve acoustic model training for speech recognition, especially when human-labeled data is scarce. By intelligently selecting automatically transcribed speech data based on local and global uncertainty, or by leveraging margin-based selection, they achieve significant improvements in speech recognition accuracy, even outperforming traditional methods when only out-of-domain data is available for initial training.
What Problem Does This Paper Address?
Training robust acoustic models (AMs) for automatic speech recognition (ASR) systems typically requires vast amounts of human-labeled speech data. However, manually transcribing speech is expensive and time-consuming. This paper tackles the challenge of "bootstrapping" AM training, meaning improving an initial model, when faced with limited human-annotated data and abundant unlabeled data that has been automatically transcribed. The core problem is how to effectively leverage this automatically transcribed data to enhance the AM without introducing too much noise or bias from transcription errors.
Methods
The authors propose two main SSL data selection methods:
- Local-Global Uncertainty Based Method: This method builds upon the idea of selecting utterances based on their uncertainty. It refines the "global entropy reduction maximization" (GERM) approach by simplifying calculations. Instead of complex global entropy reduction, it breaks down utterance uncertainty to the word level using confusion networks. Crucially, it incorporates both local uncertainty (from the current utterance's hypotheses) and global uncertainty (how the current utterance's hypotheses align with patterns found across the entire data pool). This helps select more informative samples. To mitigate redundancy, they also introduce random sampling for utterances with low local uncertainty.
- Margin Based Data Selection: This method aims to select utterances that lie close to the decision boundary of the ASR system. Directly calculating this distance is difficult for deep neural network (DNN) based ASR. The authors propose an indirect approach: they tune the language model (LM) to create a "strong" (in-domain) and "weak" (out-of-domain) LM. By comparing utterance uncertainties calculated with both LMs, they can identify samples that are more sensitive to the LM's influence, effectively selecting those near the decision boundary.
Datasets
The experiments were conducted on a Japanese far-field ASR system. The paper doesn't explicitly name specific public datasets but refers to:
- Out-domain data: Used for initial AM training when in-domain human data was unavailable.
- In-domain data: A small proportion (less than 3%) of human-annotated data used for initializing the AM in a second scenario.
- Unlabeled speech data: A large corpus (several hundred times larger than previous work) that was automatically transcribed for SSL.
Key Insights and Ideas
- Global context matters: Simply relying on the confidence of a single utterance's transcription (local uncertainty) is insufficient. Incorporating information from the entire data pool (global uncertainty) significantly improves data selection for SSL.
- Balancing reliability and informativeness: The core challenge in SSL is finding a balance. Too reliable samples don't offer new learning opportunities, while too noisy samples degrade performance. The proposed methods aim to strike this balance.
- Margin-based selection is powerful, but needs careful implementation: While selecting samples near the decision boundary is theoretically sound, its practical application in DNN-ASR requires clever workarounds like LM tuning.
- Random sampling can be a double-edged sword: While random sampling can reduce redundancy, applying it to low-uncertainty data (which might contain useful information) needs to be done carefully. The paper shows that random sampling on low-uncertainty data after local-global uncertainty selection is beneficial.
- bMMI sequence training benefits from better SSL data: Unlike previous work where sequence training didn't improve performance with automatically transcribed data, this paper shows gains, suggesting their SSL methods select higher-quality data.
Key Results
- Out-domain initialization: When no in-domain human data was available, the proposed SSL methods achieved a relative Word Error Rate (WERR) reduction of 17% compared to the initial model trained on out-domain data. The "rs_local+global" method (local-global uncertainty with random sampling on low-uncertainty data) was particularly effective, reducing the selected data size by 77% while improving performance.
- In-domain initialization: With a small amount of in-domain human data for initialization, the SSL methods yielded a relative WERR reduction of 3.7%.
- Margin-based method: When used with in-domain initialization, the margin-based method alone didn't perform well. However, when the selected data was merged with the initial supervised data, a gain was observed.
Limitations and Future Work
- Redundancy in Local-Global Uncertainty: The paper acknowledges that dropping the utterance entropy adjustment step from GERM might lead to redundant training samples, although their random sampling strategy mitigates this.
- Margin-based LM tuning: The margin-based method relies on LM tuning. While they used a strong and weak LM, the paper notes that other LM tuning techniques (like pruning) were not investigated.
- Hyperparameter tuning: The effectiveness of the proposed methods relies on hyperparameters (e.g., $\alpha$, $\beta$, $t_1$, $t_2$), which were tuned empirically.
Future work could explore more sophisticated methods for redundancy reduction in the local-global uncertainty approach and investigate alternative ways to implement margin-based selection without relying solely on LM tuning.
Summary updated