Factual Consistency Oriented Speech Recognition

Naoyuki Kanda, Takuya Yoshioka, Yang Liu

Abstract

This paper presents a novel optimization framework for automatic speech recognition (ASR) with the aim of reducing hallucinations produced by an ASR model. The proposed framework optimizes the ASR model to maximize an expected factual consistency score between ASR hypotheses and ground-truth transcriptions, where the factual consistency score is computed by a separately trained estimator. Experimental results using the AMI meeting corpus and the VoxPopuli corpus show that the ASR model trained with the proposed framework generates ASR hypotheses that have significantly higher consistency scores with ground-truth transcriptions while maintaining the word error rates close to those of cross entropy-trained ASR models. Furthermore, it is shown that training the ASR models with the proposed framework improves the speech summarization quality as measured by the factual consistency of meeting conversation summaries generated by a large language model.

Introduction: The Hallucination Problem in ASR

Standard Automatic Speech Recognition (ASR) models are typically optimized and evaluated using Word Error Rate (WER). However, WER is a semantically blind metric; it treats all word substitutions equally. A transcription that changes "I don't know" to "I dunno" is penalized the same as one that changes it to "I know," even though the latter completely reverses the meaning—a phenomenon known as a "hallucination."

As ASR models increasingly leverage powerful Language Models (LMs) to improve fluency, they become more susceptible to generating these fluent but factually ungrounded errors. This is particularly problematic for downstream Natural Language Processing (NLP) tasks. For example, if an ASR system provides a factually incorrect transcript to a summarization bot, the resulting summary will be misleading, regardless of how "accurate" the ASR system's WER might appear.

Methodology: Factual Consistency Maximization (FCM)

The authors propose a framework called Factual Consistency Maximization (FCM). Instead of just minimizing the difference between predicted and actual words (Cross-Entropy loss), they optimize the model to maximize an expected factual consistency score.

The FCM framework utilizes an external factual consistency evaluator to update the ASR model based on the semantic reliability of its N-best hypotheses.

The FCM process involves several key steps:

  1. N-Best Hypotheses: During training, the model generates several possible transcriptions (an "N-best" list) for a piece of audio.
  2. Consistency Scoring: Each hypothesis is compared against the ground-truth transcription using a separately trained factual consistency evaluator (specifically, UniEval or FactCC). These evaluators are typically based on large language models trained to detect semantic contradictions.
  3. Expected Reward: The ASR model is updated to increase the probability of hypotheses that receive high consistency scores.
  4. Safeguarding Performance: Because consistency evaluators might inadvertently reward very short, "safe" sentences, the authors use an early-stopping approach. They start with a well-trained standard model and apply a limited number of FCM updates to refine its factual reliability without degrading its ability to recognize words.

Experimental Results

The researchers evaluated their method on two major benchmarks: the AMI Meeting Corpus (approx. 100 hours) and VoxPopuli (approx. 522 hours). The FCM-trained models showed significant improvements in factual reliability across the board.

Performance comparison on the AMI-IHM dataset, showing improvements in factual consistency (UE/FCC) with stable WER.

Key Findings

  • Consistency Gains: On the AMI evaluation set, the factual consistency score (UniEval) improved from 0.787 (standard training) to 0.801 (FCM), while the WER remained nearly identical (12.6% vs 12.5%).
  • Downstream Impact: When ASR transcripts were fed into a summarization model (InstructGPT), the summaries generated from FCM transcripts were significantly more factually consistent (0.748) compared to those from standard transcripts (0.745), nearly reaching the "upper bound" score of summaries created from perfect human transcripts (0.753).
  • Robustness: The improvements were consistent across different evaluators and datasets, suggesting the model was learning general factual reliability rather than just "gaming" a specific metric.

Conclusion and Future Work

FCM provides a crucial bridge between ASR and NLP modules. By allowing the ASR model to "learn" what matters to an NLP module—factual consistency—the system becomes more reliable for real-world applications like automated meeting minutes. Future work will focus on reducing the computational overhead of generating N-best hypotheses and refining the balance between standard recognition loss and consistency loss.

Summary updated

Abstract

This paper presents a novel optimization framework for automatic speech recognition (ASR) with the aim of reducing hallucinations produced by an ASR model. The proposed framework optimizes the ASR model to maximize an expected factual consistency score between ASR hypotheses and ground-truth transcriptions, where the factual consistency score is computed by a separately trained estimator. Experimental results using the AMI meeting corpus and the VoxPopuli corpus show that the ASR model trained with the proposed framework generates ASR hypotheses that have significantly higher consistency scores with ground-truth transcriptions while maintaining the word error rates close to those of cross entropy-trained ASR models. Furthermore, it is shown that training the ASR models with the proposed framework improves the speech summarization quality as measured by the factual consistency of meeting conversation summaries generated by a large language model.
BibTeX
@misc{kanda2023factual,
  author = {Naoyuki Kanda and Takuya Yoshioka and Yang Liu},
  title = {{Factual Consistency Oriented Speech Recognition}},
  year = {2023},
  eprint = {2302.12369},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.12369},
}
Figure