Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation

Xiaoqiang Wang, Yanqing Liu, Jinyu Li, Sheng Zhao

Abstract

Automatic Speech Recognition (ASR) systems frequently struggle with domain-specific terms, such as personal names or technical jargon, which are often underrepresented in general training data. To address this, Contextual Spelling Correction (CSC) models are used as post-processors. This paper introduces CSCv3, an advanced non-autoregressive correction model that incorporates external acoustic attention and semantic-aware data augmentation. By enabling the model to "listen" to original audio features and training it on realistic error distributions, CSCv3 achieves a 20.3% relative improvement in name recall over standard biasing methods while maintaining a compact 6.2MB footprint.

Introduction

End-to-end (E2E) ASR systems often utilize "contextual biasing" to improve the recognition of niche terms like contact names or song titles. Previous state-of-the-art models, such as CSCv2, operated primarily as text-only post-processors. These models take the ASR's initial hypothesis and attempt to map it to a provided "bias list" of phrases. However, text-only correction suffers from an information bottleneck: if the ASR output is severely degraded or contains homophones, the model lacks the acoustic evidence needed to distinguish the correct term. Furthermore, standard training routines often fail to prepare models for "anti-context" scenarios, where the model must correctly decide not to apply a correction when the target term is absent from the bias list.

Architectural Innovations

CSCv3 introduces a multi-modal approach to spelling correction by integrating acoustic information directly into the correction process. The authors explored two primary architectures to bridge the gap between text and audio:

  • External Acoustic Attention (EA): In the CSCv3-EA variant, an "acoustic adapter" processes hidden features from the ASR's audio encoder. A cross-attention mechanism allows the text encoder to incorporate these acoustic cues, creating a more robust representation of the initial hypothesis.
  • Decoder Acoustic Attention (DA): The CSCv3-DA variant applies this acoustic attention within the decoder layers instead.

Empirical results indicated that incorporating acoustic information at the encoder level (EA) was more effective, suggesting that a better-informed initial representation is more valuable than attempting to resolve ambiguities during the final decoding step.

Figure 1: Comparison of the baseline CSCv2 architecture with the proposed CSCv3-EA and CSCv3-DA models, highlighting the integration of the acoustic adapter and attention mechanisms.

Semantic Aware Data Augmentation

To bridge the gap between training and real-world inference, the authors developed two key data augmentation strategies:

  1. Realistic Ref-Hyp Pairs: Using Text-to-Speech (TTS) and ASR, the researchers generated synthetic training pairs that mimic the specific phonetic and linguistic errors made by actual recognition systems.
  2. Anti-Context Training: To reduce over-correction, the model was intentionally trained on scenarios where the ground-truth phrase was removed from the bias list or replaced with similar-sounding distractors. This teaches the model to be "skeptical" and only apply corrections when there is high confidence in a match.

Experimental Results

The models were evaluated on a test set of Microsoft Teams meetings using a bias list of 1,500 phrases. Performance was measured across varying levels of "coverage"—the percentage of spoken names actually present in the bias list.

Figure 4: Performance comparison (Recall and WER) across different bias list coverage levels. CSCv3-EA consistently outperforms baselines, particularly in high-coverage scenarios.

As shown in Figure 4, CSCv3-EA achieved a significant 20.3% relative name recall gain compared to a baseline ASR system using standard Finite State Transducer (FST) biasing. Notably, in the difficult "25% coverage" scenario, CSCv3 maintained stable performance, demonstrating its robustness against false positives in sparse-context environments. At 100% coverage, the model reached a recall of 77.1%, a substantial improvement over the 75.1% achieved by the text-only CSCv2.

Efficiency and Latency

Despite the addition of acoustic processing, the student version of CSCv3 remains highly efficient at 6.2MB. While the initial latency was measured at 45.5ms per utterance, the authors implemented a caching system for bias phrase embeddings that reduced this to 20.9ms, making the system viable for real-time applications. Latency analysis revealed that the bias encoder is the primary bottleneck, accounting for 68.1% of processing time, which serves as a target for future optimization.

Conclusion

CSCv3 demonstrates that re-introducing acoustic information into the post-processing stage effectively breaks the information bottleneck of text-only correction. Combined with semantic-aware data augmentation that accounts for anti-context scenarios, the model provides a more accurate and robust solution for contextual ASR. Future work will focus on optimizing the bias encoder and diversifying training data to include more general, non-name-centric utterances.

Summary updated

Abstract

We previously proposed contextual spelling correction (CSC) to correct the output of end-to-end (E2E) automatic speech recognition (ASR) models with contextual information such as name, place, etc. Although CSC has achieved reasonable improvement in the biasing problem, there are still two drawbacks for further accuracy improvement. First, due to information limitation in text only hypothesis or weak performance of ASR model on rare domains, the CSC model may fail to correct phrases with similar pronunciation or anti-context cases where all biasing phrases are not present in the utterance. Second, there is a discrepancy between the training and inference of CSC. The bias list in training is randomly selected but in inference there may be more similarity between ground truth phrase and other phrases. To solve above limitations, in this paper we propose an improved non-autoregressive (NAR) spelling correction model for contextual biasing in E2E neural transducer-based ASR systems to improve the previous CSC model from two perspectives: Firstly, we incorporate acoustics information with an external attention as well as text hypotheses into CSC to better distinguish target phrase from dissimilar or irrelevant phrases. Secondly, we design a semantic aware data augmentation schema in training phrase to reduce the mismatch between training and inference to further boost the biasing accuracy. Experiments show that the improved method outperforms the baseline ASR+Biasing system by as much as 20.3% relative name recall gain and achieves stable improvement compared to the previous CSC method over different bias list name coverage ratio.
BibTeX
@inproceedings{wang2023improving,
  author = {Xiaoqiang Wang and Yanqing Liu and Jinyu Li and Sheng Zhao},
  title = {{Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=uAF8IO7qCa},
}
Figure