New Datasets and Controllable Iterative Data Augmentation Method for Code-switching ASR Error Correction

Zhaohong Wan, Xiaojun Wan, Wei Peng, Rongjun Li

Introduction

Automatic Speech Recognition (ASR) systems frequently struggle in multilingual contexts where speakers engage in "code-switching"—the practice of alternating between two or more languages within a single conversation (e.g., "我要 start on 我的 essay"). Correcting these errors is a critical post-processing step; however, the field faces a significant data bottleneck. High-quality, human-annotated parallel data for code-switching is scarce and expensive to produce. This paper addresses this challenge by introducing two new benchmarks and a novel data augmentation strategy.

New Benchmarks: SEAME-C and ASCEND-C

To facilitate research, the authors present two new datasets derived from existing corpora:

  • SEAME-C: Based on the SEAME corpus (Singapore/Malaysia), this dataset is characterized by high code-switching density, with 76.6% of sentences containing mixed languages.
  • ASCEND-C: Based on the ASCEND corpus (Hong Kong), this dataset is more Chinese-dominant, with code-switching appearing in 27.8% of sentences.

Analysis of these datasets reveals that "Word Selection" errors (often involving homophones) constitute the vast majority (81-85%) of ASR mistakes in code-switching scenarios, providing a clear target for correction models.

Methodology

The proposed solution involves a specialized model architecture paired with a novel data generation technique.

Sequence Editing Model

Rather than using standard sequence-to-sequence models that regenerate entire sentences, the authors employ a Sequence Editing Model based on m-BERT. This model predicts specific edit operations—keep, delete, append, or replace—for each token, making it highly efficient for error correction where the majority of the input is typically correct.

Controllable Iterative (CI) Data Augmentation

The core contribution is the Controllable Iterative (CI) data augmentation method. This process generates high-quality training pairs from monolingual text through an iterative "unfolding" process.

The unfolded process of the Controllable Iterative (CI) data augmentation method.

As shown in Figure 1, the process begins by transforming a monolingual sentence into a "clean" code-switched target. Subsequent iterations introduce synthetic ASR noise (e.g., homophone swaps). A "Confidence Bias" parameter allows researchers to control the intensity of the generated errors, while a BERTScore-based filter ensures the resulting pairs remain semantically coherent.

Experimental Results

The CI method was evaluated using the MaxMatch ($M^2$) scorer and Mixed Error Rate (MER).

Performance Gains

On the SEAME-C dataset, the CI-augmented mT5 model achieved an $F_{0.5}$ score of 26.5, significantly outperforming rule-based (25.4) and back-translation (25.5) baselines. Notably, specialized models fine-tuned on this data even outperformed large language models like ChatGPT (GPT-3.5-turbo), highlighting the value of task-specific training data.

The Iteration Bottleneck

A key finding of the study is the relationship between the number of augmentation iterations and model performance.

F0.5 score and amount of augmented data relative to the number of iterations.

As illustrated in Figure 2, while increasing iterations initially improves performance by providing more diverse training signals, the quality of generated errors begins to degrade after the 3rd iteration. This "3-iteration sweet spot" represents the optimal balance between data volume and noise quality.

Conclusion

This work demonstrates that controllable, iterative data augmentation can effectively overcome the data scarcity hurdle in code-switching ASR error correction. By combining specialized sequence editing models with high-quality synthetic data, the authors achieve state-of-the-art results that surpass general-purpose LLMs. Future work will explore extending this methodology to other language pairs and low-resource linguistic tasks.

Summary updated

Abstract

With the wide use of automatic speech recognition(ASR) systems, researchers pay more attention to the ASR error correction task to improve the quality of recognition results. In particular, ASR in bilingual or multilingual settings, namely code-switching ASR, has greater challenges and research value. In this paper, we first present code-switching ASR correction datasets obtained from solid ASR systems and automatic annotators. The datasets contain Chinese-English code-switching dialogues of bilingual speakers in Singapore, Malaysia, and Hong Kong. Based on this task, we propose a controllable iterative (CI) data augmentation method for improving the performance of mainstream ASR error correction systems. With a small amount of training data, our proposed method has the ability to iteratively produce abundant pseudo parallel data from the monolingual corpus for Chinese-English code-switching ASR correction. Results of experiments show that our method achieves the best performance compared with the rule-based, back-translation-based data augmentation methods and large language model ChatGPT.
BibTeX
@inproceedings{wan2023new,
  author = {Zhaohong Wan and Xiaojun Wan and Wei Peng and Rongjun Li},
  title = {{New Datasets and Controllable Iterative Data Augmentation Method for Code-switching ASR Error Correction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.543},
}
Figure