A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenario

Zheshu Song, Ziyang Ma, Yifan Yang, Jianheng Zhuo, Chen XiearXiv 2024 · DOI

TL;DR

This paper investigates the effectiveness of Large Language Model (LLM)-based Automatic Speech Recognition (ASR) systems compared to the popular Whisper model, particularly in low-resource languages and Mandarin-English code-switching scenarios. The study finds that LLM-based ASR offers significant improvements in low-resource settings, while Whisper-finetuned models remain superior for code-switching tasks.

What Problem Does This Paper Address?

The field of Automatic Speech Recognition (ASR) has seen remarkable progress, with models like Whisper achieving excellent performance across many languages. However, a significant challenge remains in low-resource languages, where data is scarce, and in complex scenarios like code-switching (mixing languages within a single utterance). While Large Language Models (LLMs) have demonstrated impressive capabilities in various Natural Language Processing (NLP) tasks, their application and comparative performance in these challenging ASR domains are not fully understood. This paper aims to bridge this gap by systematically comparing LLM-based ASR systems against Whisper, evaluating their strengths and weaknesses in low-resource and code-switching ASR.

Methods

The paper compares two primary ASR paradigms:

  • Whisper: This is an encoder-decoder Transformer model trained on a massive dataset for various speech tasks, including multilingual recognition and translation. It directly processes speech features and outputs text.
  • LLM-based ASR: This approach leverages a pre-trained LLM as the core component. It typically involves:
  • A speech encoder (in this study, the Whisper encoder is used) to extract speech features.
  • A linear projector to align the dimensionality of speech features with the LLM's input requirements.
  • A pre-trained LLM that takes the projected speech features, along with a prompt and the target transcript (during training), to generate the recognized text. The LLM's rich textual knowledge and reasoning abilities are hypothesized to enhance ASR performance.

For Whisper-finetune, the paper mentions applying LoRA (Low-Rank Adaptation) to the Whisper model, a parameter-efficient fine-tuning technique.

Datasets

The experiments were conducted on the following datasets:

  • GigaSpeech2: Used for general ASR tasks and for extracting low-resource language data.
  • Common Voice: Another source for low-resource language data.
  • ASRU 2019 Mandarin-English Code-Switching Challenge dataset: Specifically used for the code-switching ASR experiments.

For low-resource ASR, the paper selected Thai (th), Vietnamese (vi), Arabic (ar), and Welsh (cy), with training data ranging from 60 to 200 hours per language. For Mandarin-English code-switching, approximately 200 hours of data were used.

Key Insights and Ideas

  • LLM-based ASR shines in low-resource settings: When Whisper struggles due to limited data, LLM-based ASR systems can leverage their pre-trained knowledge to achieve significantly better performance.
  • Whisper is king for code-switching: For Mandarin-English code-switching, fine-tuned Whisper models outperform LLM-based ASR. This suggests that Whisper's direct training on diverse speech and its inherent multilingual capabilities make it more robust for this specific complex task.
  • LLM proficiency matters: The performance of an LLM-based ASR system is directly tied to the LLM's understanding and proficiency in the target language. Using LLMs specifically trained or fine-tuned for a language (e.g., Sailor for Southeast Asian languages) yields better results.

Key Results

  • Low Resource ASR: LLM-based ASR outperformed both Whisper and Whisper-finetune in three out of four low-resource languages (Thai, Vietnamese, Welsh), achieving an average Word Error Rate (WER) reduction of 12.8% compared to Whisper-finetune.
  • For Welsh, LLM-based ASR achieved a WER of 10.88%, a significant improvement over Whisper's 31.86% (and 18.06% after LoRA fine-tuning).
  • For Arabic, LLM-based ASR was better than Whisper but slightly inferior to Whisper-finetune, attributed to the LLM's limited Arabic support.
  • Mandarin-English Code-Switching ASR: LLM-based ASR achieved a mixed error rate (MER) below 8%, a 20% relative improvement over Whisper. However, Whisper-finetune still achieved a lower MER (6.12%) compared to the LLM-based ASR (7.99%).

Limitations and Future Work

  • LLM Choice is Crucial: The paper highlights that the performance of LLM-based ASR is highly dependent on the chosen LLM. While this is a strength, it also means careful selection is required, and a "one-size-fits-all" LLM might not be optimal.
  • Code-Switching Nuances: While LLM-based ASR shows promise, Whisper-finetuned models remain the state-of-the-art for Mandarin-English code-switching. Further research could explore how to better integrate LLM capabilities for such complex scenarios.
  • Future Directions: The authors hope their study will facilitate research in low-resource ASR and provide insights for the LLM-based ASR community. Future work could involve exploring more advanced LLM integration techniques, further optimizing LLM selection for specific languages, and investigating LLM-based ASR for even more challenging code-switching scenarios.

Summary updated

Abstract

Large Language Models (LLMs) have showcased exceptional performance across diverse NLP tasks, and their integration with speech encoder is rapidly emerging as a dominant trend in the Automatic Speech Recognition (ASR) field. Previous works mainly concentrated on leveraging LLMs for speech recognition in English and Chinese. However, their potential for addressing speech recognition challenges in low resource settings remains underexplored. Hence, in this work, we aim to explore the capability of LLMs in low resource ASR and Mandarin-English code switching ASR. We also evaluate and compare the recognition performance of LLM-based ASR systems against Whisper model. Extensive experiments demonstrate that LLM-based ASR yields a relative gain of 12.8\% over the Whisper model in low resource ASR while Whisper performs better in Mandarin-English code switching ASR. We hope that this study could shed light on ASR for low resource scenarios.
BibTeX
@misc{song2024comparative,
  author = {Zheshu Song and Ziyang Ma and Yifan Yang and Jianheng Zhuo and Chen Xie},
  title = {{A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenario}},
  year = {2024},
  eprint = {2412.00721},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.00721},
}
Figure