Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation

Xiaoqiang Wang, Yanqing Liu, Jinyu Li, Sheng Zhao

Abstract

Automatic Speech Recognition (ASR) systems frequently struggle with domain-specific terms, such as personal names or technical jargon, which are often underrepresented in general training data. To address this, Contextual Spelling Correction (CSC) models are used as post-processors. This paper introduces CSCv3, an advanced non-autoregressive correction model that incorporates external acoustic attention and semantic-aware data augmentation. By enabling the model to "listen" to original audio features and training it on realistic error distributions, CSCv3 achieves a 20.3% relative improvement in name recall over standard biasing methods while maintaining a compact 6.2MB footprint.

Introduction

End-to-end (E2E) ASR systems often utilize "contextual biasing" to improve the recognition of niche terms like contact names or song titles. Previous state-of-the-art models, such as CSCv2, operated primarily as text-only post-processors. These models take the ASR's initial hypothesis and attempt to map it to a provided "bias list" of phrases. However, text-only correction suffers from an information bottleneck: if the ASR output is severely degraded or contains homophones, the model lacks the acoustic evidence needed to distinguish the correct term. Furthermore, standard training routines often fail to prepare models for "anti-context" scenarios, where the model must correctly decide not to apply a correction when the target term is absent from the bias list.

Architectural Innovations

CSCv3 introduces a multi-modal approach to spelling correction by integrating acoustic information directly into the correction process. The authors explored two primary architectures to bridge the gap between text and audio:

  • External Acoustic Attention (EA): In the CSCv3-EA variant, an "acoustic adapter" processes hidden features from the ASR's audio encoder. A cross-attention mechanism allows the text encoder to incorporate these acoustic cues, creating a more robust representation of the initial hypothesis.
  • Decoder Acoustic Attention (DA): The CSCv3-DA variant applies this acoustic attention within the decoder layers instead.

Empirical results indicated that incorporating acoustic information at the encoder level (EA) was more effective, suggesting that a better-informed initial representation is more valuable than attempting to resolve ambiguities during the final decoding step.

Figure 1: Comparison of the baseline CSCv2 architecture with the proposed CSCv3-EA and CSCv3-DA models, highlighting the integration of the acoustic adapter and attention mechanisms.

Semantic Aware Data Augmentation

To bridge the gap between training and real-world inference, the authors developed two key data augmentation strategies:

  1. Realistic Ref-Hyp Pairs: Using Text-to-Speech (TTS) and ASR, the researchers generated synthetic training pairs that mimic the specific phonetic and linguistic errors made by actual recognition systems.
  2. Anti-Context Training: To reduce over-correction, the model was intentionally trained on scenarios where the ground-truth phrase was removed from the bias list or replaced with similar-sounding distractors. This teaches the model to be "skeptical" and only apply corrections when there is high confidence in a match.

Experimental Results

The models were evaluated on a test set of Microsoft Teams meetings using a bias list of 1,500 phrases. Performance was measured across varying levels of "coverage"—the percentage of spoken names actually present in the bias list.

Figure 4: Performance comparison (Recall and WER) across different bias list coverage levels. CSCv3-EA consistently outperforms baselines, particularly in high-coverage scenarios.

As shown in Figure 4, CSCv3-EA achieved a significant 20.3% relative name recall gain compared to a baseline ASR system using standard Finite State Transducer (FST) biasing. Notably, in the difficult "25% coverage" scenario, CSCv3 maintained stable performance, demonstrating its robustness against false positives in sparse-context environments. At 100% coverage, the model reached a recall of 77.1%, a substantial improvement over the 75.1% achieved by the text-only CSCv2.

Efficiency and Latency

Despite the addition of acoustic processing, the student version of CSCv3 remains highly efficient at 6.2MB. While the initial latency was measured at 45.5ms per utterance, the authors implemented a caching system for bias phrase embeddings that reduced this to 20.9ms, making the system viable for real-time applications. Latency analysis revealed that the bias encoder is the primary bottleneck, accounting for 68.1% of processing time, which serves as a target for future optimization.

Conclusion

CSCv3 demonstrates that re-introducing acoustic information into the post-processing stage effectively breaks the information bottleneck of text-only correction. Combined with semantic-aware data augmentation that accounts for anti-context scenarios, the model provides a more accurate and robust solution for contextual ASR. Future work will focus on optimizing the bias encoder and diversifying training data to include more general, non-name-centric utterances.

Summary updated

BibTeX
@inproceedings{wang2023improving,
  author = {Xiaoqiang Wang and Yanqing Liu and Jinyu Li and Sheng Zhao},
  title = {{Improving Contextual Spelling Correction by External Acoustics Attention and Semantic Aware Data Augmentation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=uAF8IO7qCa},
}
Figure