TL;DR
This paper tackles the problem of overcorrection in Automatic Speech Recognition (ASR) error correction (EC) systems, particularly in out-of-domain (OOD) scenarios. They propose a novel data filtering method based on two criteria: linguistic acceptability and inferability, which significantly reduces overcorrection and improves ASR robustness.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems are increasingly powerful, but they still make errors. Error Correction (EC) systems, often built on Large Language Models (LLMs), aim to fix these ASR errors. Typically, EC training data is created by pairing ASR hypotheses (noisy outputs) with their correct transcriptions (gold references). However, this process can introduce "noisy" pairs where the "correction" is actually unnecessary, incorrect, or uninferable. This noise can lead to overcorrection, a common problem where EC systems make unwarranted changes, degrading performance, especially when encountering speech from domains not seen during training (Out-of-Domain or OOD). Existing methods for filtering noisy data, like simply discarding pairs with large edit distances, are insufficient. This paper addresses this by proposing a more sophisticated data filtering approach to create cleaner, more robust EC training data.
Methods
The core of this paper is a novel data filtering strategy for EC training data. They introduce two fundamental criteria for high-quality EC pairs:
- Linguistic Acceptability: The corrected target (WT) should be linguistically more acceptable than the source (WS). This is quantified using a language model's likelihood ratio: $P(W_T) / P(W_S) \ge C_1$.
- Inferability: The corrected target (WT) should be inferable from the available context, primarily the source phonemes ($W^P$). This is measured using a phoneme-based EC model's likelihood ratio: $PEC(W_T | W^P) / PEC(W_S | W^P) \ge C_2$.
Pairs that fail to meet these criteria are considered "noisy" and are handled conservatively by replacing the target with the source ($W_T \rightarrow W_S$) during training, rather than discarding them. This encourages the EC model to be more cautious and avoid making corrections on ambiguous or incorrect examples. The authors also incorporate source phonemes as an additional input to the EC model, which can help in resolving phonetic confusions.
Datasets
The paper uses internal Japanese ASR training data for their experiments. For evaluation, they utilize a suite of 21 internal benchmarks covering various domains. These benchmarks are treated as Out-of-Domain (OOD) settings, meaning the EC models are not specifically adapted to them.
Key Insights and Ideas
- Noisy data is a significant problem: A substantial portion of automatically generated EC training data is noisy, leading to overcorrection and degraded performance, especially in OOD scenarios.
- Edit distance is a poor proxy for inferability: Simple filtering based on edit distance is insufficient and can even discard useful data or fail to identify truly uninferable corrections.
- Conservative filtering is key: Instead of discarding noisy pairs, the proposed method trains the model to be conservative by treating them as if no correction is needed. This is more effective than simply removing them.
- Linguistic acceptability and inferability are crucial: These two criteria effectively capture different types of noise and are essential for building robust EC models.
- Phonetic information aids EC: Including source phonemes as input to the EC model helps it better understand and correct ASR errors.
Key Results
The proposed conservative data filtering method, combining both criteria (C1+C2), significantly improved ASR results:
- Reduced Overcorrection: The frequency of corrections was drastically reduced (e.g., from 22.1% to 13.9% on average with Swallow-Mistral 7B).
- Improved CER: The Character Error Rate (CER) was reduced in 71.4% (15/21) of the test sets compared to the original ASR.
- Enhanced OOD Robustness: The approach demonstrated significant improvements in challenging OOD settings, outperforming the original ASR in a majority of test cases.
- Generalizability: The method showed similar positive results when applied to a different Japanese LLM (Sarashina-2 7B), indicating its generalizability.
- Comparison to Edit Distance Filtering: The proposed method significantly outperformed filtering based on maximum edit distance, which failed to improve CER and hardly reduced the correction rate.
Limitations and Future Work
- Threshold Tuning: The paper uses default thresholds ($C_1=C_2=1$) but notes that tuning these thresholds could potentially improve performance on specific domains or for borderline quality pairs.
- LLM Power: The authors suggest that as LLMs become more powerful and accurate, the effectiveness of their data filtering method is likely to increase.
- Other Data Quality Factors: Future work could explore controlling for other important data quality factors like diversity and representativeness to further enhance EC robustness.
- General-Purpose EC Models: The ultimate goal is to develop general-purpose EC models applicable to any domain without requiring domain-specific adaptation.
Summary updated