Abstract
Automatic Speech Recognition (ASR) systems frequently encounter errors in noisy environments or with complex linguistic contexts. While Large Language Models (LLMs) are effective for post-editing these errors, the standard "Full Rewrite" approach is computationally inefficient. We present Context-Enhanced Granular Edit Representation (CEGER), a framework that trains LLMs to generate structured editing commands—such as insertions, deletions, and replacements—rather than entire sentences. CEGER achieves state-of-the-art Word Error Rate (WER) reductions (up to 60.6%) while operating 2.5x faster than traditional methods by significantly reducing output redundancy.
Introduction
Despite advancements in ASR, post-editing remains a critical step for high-accuracy applications. The prevailing method involves an LLM reading noisy ASR output and generating a corrected version from scratch. However, since the majority of ASR output is often correct, this "Full Rewrite" approach wastes significant compute resources. Existing "compact" representations attempt to address this by only generating changes, but they often lack the necessary context or precision, leading to ambiguity.
CEGER bridges this gap by combining the accuracy of full rewrites with the efficiency of compact representations. It uses a precise grammar of word-level operations to guide the LLM in making surgical corrections.
Methodology: The CEGER Framework
The CEGER framework consists of a fine-tuned LLM that acts as a command generator and a deterministic expansion module that applies these commands to the original text.
The Command Grammar
The LLM is trained to output a sequence of four structured commands that describe the transformation from noisy ASR to ground truth:
[MOVE_FORWARD n]: Retains the next $n$ words without changes.[DELETE n]: Removes the next $n$ words.[INSERT 'text']: Adds new words at the current position.[REPLACE n WITH 'text']: Substitutes $n$ words with a new sequence.
Pipeline and Training
To train the model, the authors use the Levenshtein algorithm to convert ASR-transcript pairs into optimal CEGER command sequences. A PaLM 2 Otter model is then fine-tuned on this data. During inference, the LLM predicts the command sequence, and the Expansion Module executes them to reconstruct the final text. This deterministic execution phase eliminates the risk of "hallucinations" common in standard text generation.
Experimental Setup
The framework was evaluated using the LibriSpeech dataset. Initial ASR outputs were generated using a frozen Google Universal Speech Model (USM). Performance was measured across two subsets: test-clean (high-quality audio) and test-other (challenging audio).
Results and Discussion
CEGER demonstrated superior performance across all primary metrics compared to full rewrites and other compact baselines.
Accuracy and Efficiency
As shown in the results table, CEGER achieved the lowest Word Error Rate (WER) on both datasets. On test-clean, it reduced the initial ASR WER from 6.6% to 2.6%, representing a 60.6% improvement.
Crucially, CEGER is significantly more efficient. It recorded an average latency of 385ms per utterance, making it approximately 2.5x faster than the Full Rewrite method (980ms). This speedup is largely due to the reduced output length; CEGER's average output was only 10-11 tokens, compared to 18-20 for full rewrites. Analysis showed that [MOVE_FORWARD] was the most frequent command, allowing the model to "skip" correct text and focus compute on errors.
Human Evaluation
In side-by-side human evaluations, CEGER's outputs were preferred in over 65% of cases. Annotators noted that the structured approach led to better fluency and more faithful corrections compared to baseline methods.
Conclusion
CEGER provides a robust solution for efficient ASR post-editing by leveraging granular edit commands. While highly effective, the model still faces challenges with subtle semantic errors where the ASR output is grammatically correct but contextually inappropriate. Future work will explore integrating external knowledge graphs and more targeted training to resolve these fine-grained ambiguities.
Summary updated