Lattice Re-Scoring During Manual Editing for Automatic Error Correction of ASR Transcripts

Anna V. Rúnarsdóttir, Inga R. Helgadóttir, Jón GuðnasonInterspeech 2019

TL;DR

This paper introduces a novel method to automatically correct downstream errors in Automatic Speech Recognition (ASR) transcripts by re-scoring the ASR's word lattice after a human corrects the first error in a sentence. This approach significantly reduces sentence error rates, particularly for sentences with multiple initial errors, by leveraging the lattice's alternative hypotheses.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are increasingly used for transcribing speech into text for various applications, from official records to publications. However, even the most advanced ASR systems are prone to errors that can alter the meaning of the transcribed text. Traditionally, human editors meticulously review and correct these ASR outputs, a process that can be time-consuming and labor-intensive. This paper addresses the challenge of reducing this manual post-editing effort by proposing a method to automatically correct subsequent errors in a sentence once the initial error has been identified and corrected by a human.

Methods

The core of the proposed method lies in lattice re-scoring with search updating. When a human editor corrects the first error in an ASR transcript, the system uses this correction to re-calculate the most probable path through the ASR's word lattice. A word lattice is a data structure that represents multiple possible transcriptions of an utterance, each with an associated cost.

The process involves:

  1. Manual Correction of the First Error: A human editor identifies and corrects the first error in a sentence.
  2. Lattice Traversal and Path Re-computation: The system then searches the word lattice for paths that incorporate this manual correction. It performs a depth-first search to find all valid paths that start with the corrected sequence.
  3. Re-scoring: The system re-scores these paths, effectively finding a new "best" hypothesis for the entire utterance that is consistent with the initial manual correction. This is achieved by forcing the search to go through the corrected word.

The paper utilizes Kaldi's implementation of weighted finite-state transducers (WFSTs) for representing word lattices and employs the Bellman-Ford algorithm for finding the shortest (lowest cost) path.

Datasets

  • Training Data: Approximately 100 hours (40,000 utterances) of parliamentary speeches from the Althingi's speech recognition corpus.
  • Test Data: A combination of the development and evaluation sets from Althingi, consisting of 8,545 utterances (approximately 22 hours of acoustic data). The average utterance length in the test set is 22 words.

The paper also references existing Icelandic language resources like the Malromur acoustic database and the Risamálheild text corpus, which were instrumental in developing the ASR system.

Key Insights and Ideas

  • Leveraging the Lattice for Downstream Correction: The most significant insight is that the word lattice, often used for initial decoding, can be re-purposed after a manual edit to correct subsequent errors. This moves beyond traditional lattice re-scoring which typically occurs before human intervention.
  • The Power of the First Correction: Correcting just the first error in a sentence can have a cascading positive effect, significantly improving the accuracy of the rest of the sentence by guiding the re-scoring process.
  • Trade-off Favors Automation: The paper demonstrates that the benefits of automatically fixing errors through re-scoring heavily outweigh the risk of introducing new errors, making it a valuable addition to the transcription pipeline.
  • Impact on Sentence Error Rate (SER): The method shows a substantial reduction in SER, particularly for sentences that initially contained multiple errors, highlighting its effectiveness in complex cases.

Key Results

The lattice re-scoring method was applied to 4,807 utterances in the test set (excluding those already correct or where the correction was not found in the lattice).

  • Overall Error Reduction: The total number of errors in the test set was reduced by 5.5% after applying the lattice re-scoring method.
  • Correction of Subsequent Errors: 32.34% of errors immediately following a manually corrected first error were automatically fixed.
  • Sentence Error Rate (SER) Improvement:
    • For utterances with two errors, SER dropped from 100% to 82.77% (a reduction of 17.23%).
  • For utterances with three errors, SER dropped from 100% to 95.88% (a reduction of 4.12%).
  • New Errors Introduced: New errors were introduced in 17.78% of the utterances that had two or more errors initially.
  • Out-of-Vocabulary (OOV) Words: 8.2% of uncorrected errors were due to out-of-vocabulary words, which the system cannot correct as they are not in its lexicon.

Limitations and Future Work

  • Lexicon Dependency: The method's effectiveness is limited by the ASR system's lexicon. If a corrected word is not in the lexicon, the system cannot find a path for it, and thus cannot correct subsequent errors.
  • Integration into Editors: The authors acknowledge that integrating this lattice re-scoring capability into a practical text editor for ASR systems needs further investigation.
  • Measuring Editing Time: Future work should focus on measuring whether this method actually decreases human editing time in real-world scenarios or if the new suggestions might sometimes slow down editors.
  • Execution Time and Storage: The computational cost and storage requirements for Kaldi lattices need to be considered for practical deployment.
  • Distance of Errors: The paper notes that an analysis of how the distance between the first error and subsequent errors affects the benefits of the proposed method has not been performed.

Summary updated

BibTeX
@inproceedings{runarsdottir2019lattice,
  author = {Anna V. Rúnarsdóttir and Inga R. Helgadóttir and Jón Guðnason},
  title = {{Lattice Re-Scoring During Manual Editing for Automatic Error Correction of ASR Transcripts}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure