Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Study

Zeping Min, Jinbo Wang

TL;DR

This paper investigates whether Large Language Models (LLMs) like ChatGPT and GPT-4 can improve Automatic Speech Recognition (ASR) accuracy by correcting transcription errors using their in-context learning abilities. Despite extensive experimentation with various LLM models, instructions, and few-shot settings, the study found that directly applying LLMs for ASR error correction often increases the Word Error Rate (WER), highlighting significant challenges in this application area.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are becoming increasingly ubiquitous, but they still struggle with real-world challenges like background noise, diverse speaker accents, and complex linguistic contexts. These issues lead to transcription errors, limiting the effectiveness of ASR. While Large Language Models (LLMs) have shown remarkable capabilities in various Natural Language Processing (NLP) tasks, their potential to directly enhance ASR performance by correcting transcription errors using their "in-context learning" ability (i.e., learning from examples provided in the prompt without explicit fine-tuning) is an emerging area of research that this paper aims to explore. The core problem is to determine if LLMs can reliably and effectively fix ASR mistakes.

Methods

The researchers designed an empirical study to evaluate the effectiveness of LLMs for ASR error correction. Their methodology involved:

  • Prompt Engineering: Crafting specific instructions for the LLMs to act as transcription checkers and correct errors. They experimented with four different instruction sets, ranging from simple commands to more detailed explanations of error types.
  • In-Context Learning: Providing the LLMs with examples of ASR transcriptions and their corrected versions within the prompt. They varied the number of examples (1-shot, 2-shot, 3-shot).
  • Iterative Correction: Exploring the impact of allowing the LLM multiple attempts (1, 3, or 5) to correct each transcription sentence, with the best correction chosen based on the lowest Word Error Rate (WER).
  • Model Variations: Testing different versions of LLMs, including GPT-3.5 (various context window sizes) and GPT-4.
  • ASR Baseline: Using a standard hybrid CTC/attention ASR model trained on the respective datasets as a baseline for comparison.

Datasets

The study utilized two well-established ASR datasets:

  • Aishell-1: A Chinese language dataset with 178 hours of audio, known for its high-quality manual transcriptions.
  • LibriSpeech: An English language dataset comprising approximately 1000 hours of audio extracted from audiobooks, further categorized into "clean" and "other" (more challenging) subsets.

These datasets are commonly used benchmarks in ASR research.

Key Insights and Ideas

  • LLMs Don't Automatically Improve ASR Correction: The most striking finding is that directly applying LLMs' in-context learning for ASR error correction often worsens performance, leading to higher Word Error Rates (WERs) compared to the original ASR output. This challenges the initial assumption that LLMs' general language understanding would translate directly to effective ASR correction.
  • Complexity of Speech Data: Speech recognition errors are nuanced and context-dependent. LLMs, despite their linguistic prowess, may struggle to accurately interpret and correct these errors without a deeper understanding of the acoustic and phonetic aspects of speech, which are not directly encoded in text transcriptions.
  • Instruction Detail Matters, But Not Enough: While more detailed instructions generally led to slightly better (or less worse) results, even the most carefully crafted prompts and extensive few-shot examples could not overcome the fundamental limitations.
  • GPT-4 vs. GPT-3.5: While GPT-4 is generally more capable, the study found that even GPT-4, when applied to this task, did not yield satisfactory results, reinforcing the idea that the problem lies in the task itself rather than just the LLM's raw capability.
  • "Best" Correction is Unrealistic: The experimental approach of selecting the "best" correction based on the lowest WER is not practical in real-world scenarios, as the ground truth is unknown. This highlights the difficulty of evaluating and deploying such systems.

Key Results

  • Increased WER: In almost all experimental conditions, using LLMs for ASR correction resulted in a higher WER than the original ASR output. For instance, on the LibriSpeech clean dataset, the baseline WER was 3.35%, while LLM corrections often pushed this to over 47% (e.g., Table 3, GPT-3.5-turbo-4k-0301).
  • Specific Examples of Errors: The paper provides illustrative examples where LLMs introduced new errors or misinterpreted words, such as changing "STUFFED INTO YOU HIS BELLY" to "STUFF IT INTO HIS BELLY" or "FRESH NELLIE" to "EXPRESS DELI," demonstrating a lack of semantic preservation and factual accuracy.
  • Varying Settings:
    • LLM Models: GPT-3.5-turbo-4k-0613 and GPT-3.5-turbo-16k-0613 performed better than GPT-3.5-turbo-4k-0301, but still significantly worse than the baseline without LLM.
  • Instructions: Instruction 4, the most detailed, generally performed better than simpler instructions, but still resulted in higher WERs.
  • Shots: Increasing the number of examples (shots) showed a trend towards improvement, but never reached satisfactory WER levels.
  • Attempts: Even with up to 5 attempts per sentence, the LLM-corrected outputs still introduced more errors.

Limitations and Future Work

  • Limitations:
    • The primary limitation is that directly applying LLMs' in-context learning for ASR error correction is currently ineffective and often detrimental, leading to higher WERs.
  • The study acknowledges that selecting the output with the lowest WER is not practical in real-world scenarios.
  • The research focused on specific LLM models (GPT-3.5 and GPT-4) and may not generalize to all LLMs.
  • Future Work:
    • The authors suggest that further research and development are needed to optimize the use of LLMs in ASR.
  • As LLMs continue to evolve, their capabilities might be harnessed more effectively in the future.
  • The study aims to inspire further research to refine and improve LLM applications in ASR technology.

Summary updated

Abstract

This paper explores the integration of Large Language Models (LLMs) into Automatic Speech Recognition (ASR) systems to improve transcription accuracy. The increasing sophistication of LLMs, with their in-context learning capabilities and instruction-following behavior, has drawn significant attention in the field of Natural Language Processing (NLP). Our primary focus is to investigate the potential of using an LLM’s in-context learning capabilities to enhance the performance of ASR systems, which currently face challenges such as ambient noise, speaker accents, and complex linguistic contexts. We designed a study using the Aishell-1 and LibriSpeech datasets, with ChatGPT and GPT-4 serving as benchmarks for LLM capabilities. Unfortunately, our initial experiments did not yield promising results, indicating the complexity of leveraging LLM’s in-context learning for ASR applications. Despite further exploration with varied settings and models, the corrected sentences from the LLMs frequently resulted in higher Word Error Rates (WER), demonstrating the limitations of LLMs in speech applications. This paper provides a detailed overview of these experiments, their results, and implications, establishing that using LLMs’ in-context learning capabilities to correct potential errors in speech recognition transcriptions is still a challenging task at the current stage.
BibTeX
@inproceedings{min2023exploring,
  author = {Zeping Min and Jinbo Wang},
  title = {{Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Study}},
  booktitle = {ICONIP},
  year = {2023},
  url = {https://openreview.net/forum?id=1GBCe7IdyL},
}

Loading related content...

Figure