HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

Chen Chen 0075, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, Eng Siong Chng

Introduction

Automatic Speech Recognition (ASR) has seen significant advancements with models like Whisper and WavLM. However, performance often degrades in challenging conditions such as high background noise, diverse accents, or domain-specific terminology. The traditional solution, N-best rescoring, involves selecting the most linguistically plausible candidate from a list of top-N hypotheses generated by the ASR engine.

A critical limitation of this approach is its reliance on the presence of the correct transcription within the N-best list. If the ground truth is absent, rescoring merely selects the "least incorrect" option. This paper introduces HyPoradise (HP), a benchmark that shifts the paradigm from discriminative rescoring to generative error correction. By leveraging the linguistic capabilities of Large Language Models (LLMs), the proposed framework synthesizes the correct transcription from the collective information present across all N-best hypotheses.

Figure 2: Overview of the ASR pipeline and the motivation for generative correction. Case (i) shows instances where a lower-ranked hypothesis is better than the first, while Case (ii) illustrates how correct tokens can be recovered from the N-best list even if they are missing from the top result.

The HyPoradise Dataset

The core of this work is the HyPoradise dataset, comprising over 334,000 pairs of N-best hypotheses and their corresponding ground-truth transcriptions. The dataset is characterized by its high diversity, spanning nine distinct speech domains:

  • Clean Speech: LibriSpeech (audiobooks).
  • Noisy Environments: CHiME-4 (street, cafe, and bus noise).
  • Accented Speech: CommonVoice (various regional accents).
  • Specialized Contexts: Wall Street Journal (business), ATIS (airline information), and CORAAL (African American Language).

Hypotheses were generated using state-of-the-art models (WavLM and Whisper-Large), ensuring the benchmark accurately reflects the error patterns of modern ASR systems.

Methodology: Hypotheses-to-Transcription (H2T)

The authors propose the Hypotheses-to-Transcription (H2T) framework, which treats error correction as a sequence-to-sequence task. The entire N-best list is provided as input to an LLM, which then generates the final transcription. Three primary implementation strategies were evaluated:

  1. In-Context Learning (Zero/Few-shot): Utilizing GPT-3.5 with Task-Activating Prompting (TAP). The model is provided with a few demonstration examples to guide its generation without updating internal weights.
  2. Fine-tuning (H2T-ft): Training smaller models like T5-large specifically on the HP dataset to map noisy hypotheses to clean text.
  3. Parameter-Efficient Fine-tuning (H2T-LoRA): Applying Low-Rank Adaptation (LoRA) to a LLaMA-13B model. This allows for domain adaptation with significantly lower computational overhead than full fine-tuning.
Figure 6: (a) The high-level architecture of the correction model taking prompts and N-best lists as input. (b) The LoRA fine-tuning mechanism used to adapt pre-trained weights efficiently.

Key Insights and Results

The generative approach demonstrates several advantages over traditional rescoring:

  • Surpassing the N-best Oracle: Generative LLMs can achieve a lower Word Error Rate (WER) than the theoretical limit of picking the single best candidate from the N-best list (the "Oracle" WER). This is possible because the model can combine correct tokens from different hypotheses.
  • Linguistic Inference: LLMs excel at correcting "specious tokens"—phonetically similar but contextually incorrect words. For instance, an LLM can infer the correct name "Sinopec" from a phonetic error like "xinnepec" by analyzing the surrounding context of the petrochemical industry.
  • Mining the N-best List: Analysis shows a 34% probability that a correct token missing from the 1st-best hypothesis exists elsewhere in the top 20 list. Generative models are uniquely capable of extracting this information.

Empirical results highlight substantial improvements. On the WSJ dataset, H2T-LoRA reduced WER from 4.5% to 2.2% (a 50% relative improvement). On the noisy CHiME-4 dataset, WER dropped from 11.1% to 6.6%. Furthermore, the corrected text improved downstream Spoken Language Intent Detection accuracy from 94.9% to 97.4%.

Conclusion and Future Directions

While HyPoradise establishes a powerful new baseline, challenges remain. The H2T-ft approach occasionally suffers from overfitting on long utterances or unfamiliar domains. Additionally, the current models are "acoustically blind," relying solely on text without access to original audio or ASR confidence scores. Future work may involve integrating these acoustic cues to help the LLM better distinguish between reliable and unreliable segments of the hypotheses. Finally, the computational cost of large-scale LLM inference remains a barrier to real-time deployment.

Summary updated

BibTeX
@inproceedings{chen2023hyporadise,
  author = {Chen Chen and Yuchen Hu and Chao-Han Huck Yang and Sabato Marco Siniscalchi and Pin-Yu Chen and Eng Siong Chng},
  title = {{HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=XdyaBlWiZX},
}
Figure