Abstract
Automatic Speech Recognition (ASR) systems frequently struggle with domain-specific named entities (NEs), particularly when the transcription results in "form-different" errors where the output text bears little visual resemblance to the target entity. This paper presents a novel "Generative Annotation" framework that combines speech-sound retrieval with a generative labeling model. By identifying errors as an annotation task rather than a direct translation task, the proposed method significantly improves named entity recall and reduces character error rates across challenging datasets.
1. Introduction
Named Entity Correction (NEC) is a critical post-processing step for Automatic Speech Recognition (ASR). While modern ASR systems are highly capable, they often fail on niche terms, brand names, and technical jargon. Traditional NEC approaches typically rely on Phonetic-level Edit Distance (PED-NEC), which searches for candidate entities based on the phonetic similarity of the transcribed text.
However, a significant bottleneck exists: "form-difference." In many cases, an ASR system might produce a string of characters that sounds similar to the target audio but looks entirely different in text (e.g., transcribing the English loanword "ChatGPT" into the Chinese characters "切特GPD"). Because PED-NEC relies on the corrupted text for retrieval, it often fails to find the correct candidate.
2. The Generative Annotation Framework
To address the limitations of text-based retrieval, we propose a two-stage method: Speech-Sound Retrieval (SS) followed by Generative Labeling (GL).
2.1 Speech-Sound Retrieval (SS)
Instead of using the potentially erroneous ASR transcript as the query, the SS module utilizes the original audio signal. A datastore is constructed where entities are represented by acoustic features generated via Text-to-Speech (TTS) and a CNN-augmented encoder. During inference, the system retrieves the top $K$ candidates that are acoustically similar to the input speech, bypassing the "form-difference" trap.
2.2 Generative Labeling (GL)
The core innovation of this research is treating correction as an annotation task. A generative decoder (based on the Whisper architecture) is provided with a prompt containing the retrieved candidates and the original ASR transcript. Rather than rewriting the entire sentence, the model is trained to "label" or identify the specific segments in the transcript that are incorrect.
2.3 Replacement and Rejection
Once the model identifies the erroneous span in the transcript (e.g., identifying that "切特GPD" is the error corresponding to the candidate "ChatGPT"), the system replaces it with the correct entity. Crucially, the model includes an "Implicit Rejection" mechanism: if no candidate matches the transcript context, the model outputs an empty label, preventing false corrections and hallucinations.
3. Experimental Evaluation
The framework was evaluated using three distinct datasets:
- AISHELL: A standard Mandarin ASR dataset used for baseline training, utilizing a subset of 54,129 entities.
- Word Form Variation Set: A curated set of 50 entities specifically chosen for high form-difference between ASR output and ground truth.
- BuzzWord Set: A challenging collection of 1,500 segments featuring 2023-2024 "buzzwords," loanwords, and alphanumeric entities that are notoriously difficult for standard ASR engines.
4. Results and Discussion
The proposed SS+GL method demonstrated substantial improvements over traditional PED-NEC baselines across all metrics.
4.1 Performance Gains
On the BuzzWord Set, NE-Recall (the ability to correctly identify and fix entities) increased from 61.82% using PED-NEC to 87.47% using SS+GL. Furthermore, on the Word Form Variation set, the Character Error Rate (CER) was reduced from 17.60% to 11.45%.
4.2 Cross-System Generalizability
The authors tested their method as a post-processor for high-performing commercial ASR engines, including iFlytek and Amazon. Even on these optimized systems, the Generative Annotation method improved NE-Recall by 35% to 70% on the BuzzWord set, proving its utility as a plug-and-play enhancement for existing ASR pipelines.
4.3 Contextual Intelligence
The success of the model is largely attributed to its "form-insensitive" nature and its use of surrounding language context. By analyzing the whole sentence, the generative model can distinguish between phonetically similar candidates (e.g., distinguishing between a person's name and a language) more effectively than simple distance-based metrics.
5. Conclusion and Future Work
Generative Annotation offers a robust solution for the "form-difference" problem in ASR. By shifting from text-based retrieval to audio-based retrieval and employing a generative labeling strategy, the system achieves high precision in entity correction. While the current implementation adds a post-processing step that may impact latency, future research will focus on optimizing the pipeline for real-time applications and scaling the retrieval datastore using vector search engines.
Summary updated