In-context Learning Demonstration Generation with Text Distillation

Yixuan Tang, Anthony K. H. Tung, Zilan Zhou, Cheng Deng2024 · DOI

TL;DR

Traditional speech recognition often struggles with "out-of-context" errors, where a model mishears a word that would be obvious if it knew the surrounding topic. This paper proposes a two-stage framework that moves beyond simply re-ranking ASR guesses; instead, it uses a Large Language Model (LLM) to regenerate the transcript by synthesizing information from multiple ASR hypotheses and the broader text context, achieving up to a 45.9% relative improvement in accuracy.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) has improved significantly, but "contextualized ASR"—recognizing speech within a specific setting like a lecture or a reading passage—remains a hurdle.

The authors identify a major flaw in the standard "second-pass rescoring" paradigm. Usually, an ASR system generates an "$n$-best" list of candidate transcriptions, and a second model simply picks the most likely one. However, if the correct word isn't in that $n$-best list at all (a common occurrence with rare words or heavy accents), the system is guaranteed to fail. This paper addresses this "error propagation" by shifting from discriminative rescoring (picking from a list) to generative re-transcription (writing a new version based on clues).

Methods

The researchers introduce a two-stage pipeline:

  1. First Pass (Hypothesis Generation): They use OpenAI’s Whisper model to process raw audio and generate a list of $n$-best hypotheses (e.g., the top 5 or 10 guesses).
  2. Second Pass (LLM Regeneration): They feed these $n$-best hypotheses, along with a "context paragraph" (the text the speaker is referring to), into a generative LLM (Mistral-7B-Instruct).

The LLM doesn't just pick one guess; it looks at the phonetics suggested by the ASR guesses and reconciles them with the semantic cues in the context. The authors explored two ways to implement this:

  • Zero-shot Prompting: Giving the LLM a carefully crafted instruction to "act as an ASR corrector" without any specific training.
  • LoRA Fine-tuning: Using Low-Rank Adaptation to efficiently fine-tune the LLM on a small set of transcript pairs, teaching it to better discern subtle contextual nuances.

Datasets

The study utilizes the SQuAD-SRC (Spoken Reading Comprehension) dataset.

  • Content: Over 98,000 spoken questions based on Wikipedia passages.
  • Diversity: Features 24 speakers from six different countries (US, UK, China, India, Japan, and Thailand), providing a rigorous test for multi-accent robustness.
  • Context: Crucially, each audio snippet is paired with the original text passage, providing the "ground truth" context needed for the LLM to perform its magic.

Key Insights and Ideas

  • The "Aha" Moment - Regeneration vs. Rescoring: The most significant insight is that an LLM can "hallucinate" the correct word back into existence. If the ASR hears "sugar bowl" and "soup bowl," but the context is about American football, the LLM can infer the speaker actually said "Super Bowl," even if that exact phrase never appeared in the ASR's top guesses.
  • Clues in the $n$-best List: Even if the $n$-best hypotheses are all wrong, they contain phonetic "clues." By looking at all of them simultaneously, the LLM gets a better "sketch" of what the audio sounded like than it would from just the single best guess.
  • Data Efficiency: The authors found that fine-tuning with as few as 100 examples (0.2 hours of speech) led to massive gains. This suggests that LLMs already "know" how to do this; they just need a tiny bit of steering to apply that knowledge to ASR correction.

Key Results

The performance was measured using Word Error Rate (WER), where lower is better.

  • Baseline (Whisper-tiny): 24.2% WER.
  • Zero-shot Prompting (n=10): Reduced WER to 20.9% (a 13.6% relative improvement).
  • LoRA Fine-tuned (2000 examples): Slashed WER to 13.1% (a massive 45.9% relative improvement).
  • Scaling Effects: Performance consistently improved as the number of $n$-best hypotheses increased from 1 to 10, proving that more "clues" help the LLM make better decisions.
  • Model Comparison: A fine-tuned Whisper-tiny model (the smallest version) actually outperformed the original Whisper-medium (a much larger model) when using this framework.

Limitations and Future Work

  • Dependency on Context: This method requires a relevant text passage to be available. It works great for lectures, meetings with agendas, or reading tasks, but might be less effective for spontaneous, open-domain conversation where no reference text exists.
  • Computational Overhead: Running a 7-billion parameter LLM for every sentence of speech is computationally expensive compared to traditional ASR.
  • Future Directions: The authors plan to explore how this generalizes across different domains and languages. They also suggest incorporating "domain information" (like a specialized dictionary) directly into the prompt to help with highly technical jargon.

Summary updated

Abstract

In-context learning (ICL), a paradigm derived from large language models (LLMs), holds significant promise but is notably sensitive to the choice of input demonstrations. While numerous methodologies have been developed to select the optimal demonstrations from existing datasets, our work alternatively proposes to generate representative demonstrations through a Distillation-based Demonstration Generation (DDG) framework. Specifically, our approach aims to generate demonstrations that encapsulate the essential attributes of the target dataset. Rather than optimizing these demonstrations directly, we design a generative model and try to refine it by minimizing the discrepancies between the calculative models trained on generated demonstrations and the original datasets respectively. Additionally, we leverage a teacher-student framework to stabilize the training process and improve the quality of the synthesized samples. Extensive experiments conducted across ten prevalent text datasets demonstrate that our DDG method substantially outperforms existing state-of-the-art methodologies. Our code will be available at https://github.com/wwyq1/DDG.
BibTeX
@misc{tang2024context,
  author = {Yixuan Tang and Anthony K. H. Tung and Zilan Zhou and Cheng Deng},
  title = {{In-context Learning Demonstration Generation with Text Distillation}},
  year = {2024},
  doi = {10.24963/ijcai.2024/716},
}
Figure