Abstract
Prompts are crucial to large language models as they provide context information such as topic or logical relationships. Inspired by this, we propose PromptASR, a framework that integrates prompts in end-to-end automatic speech recognition (E2E ASR) systems to achieve contextualized ASR with controllable style of transcriptions. Specifically, a dedicated text encoder encodes the text prompts and the encodings are injected into the speech encoder by cross-attending the features from two modalities. When using the ground truth text from preceding utterances as content prompt, the proposed system achieves 21.9% and 6.8% relative word error rate reductions on a book reading dataset and an in-house dataset compared to a baseline ASR system. The system can also take word-level biasing lists as prompt to improve recognition accuracy on rare words. An additional style prompt can be given to the text encoder and guide the ASR system to output different styles of transcriptions. The code is available at icefall.
Introduction
Standard Automatic Speech Recognition (ASR) systems typically process audio in isolation, transcribing individual snippets without awareness of the broader conversational or narrative context. This "context blindness" often leads to errors in transcribing ambiguous words or specialized terminology. Furthermore, most ASR models are rigid in their output, producing normalized text that requires separate, often brittle, post-processing steps like Inverse Text Normalization (ITN) to add casing and punctuation.
PromptASR addresses these challenges by introducing a unified framework for contextualized ASR. By leveraging the concept of "prompting" from Large Language Models (LLMs), PromptASR can incorporate conversational history, rare word lists, and stylistic instructions directly into the recognition process.
Methodology
The PromptASR framework utilizes a Transducer-based architecture enhanced by a dual-encoder setup. The system consists of a standard speech encoder (Zipformer) and a dedicated, pretrained text encoder (BERT).
As illustrated in the architecture diagram, text prompts are processed by the text encoder to generate prompt embeddings. These embeddings are then injected into the speech encoder through a cross-attention mechanism. In each layer of the speech encoder, acoustic features "look at" the text embeddings to extract relevant contextual information. To ensure the model remains robust and does not hallucinate words based solely on prompts, the authors employed "prompt dropout" and simulated irrelevant prompts during training.
Prompting Strategies
PromptASR supports two primary types of prompts:
- Content Prompts: These provide semantic context, such as the ground-truth text of preceding utterances or a "biasing list" of rare words (e.g., technical jargon or specific names) that the model should prioritize.
- Style Prompts: These act as instructions for the output format. For example, a prompt like "Mixed-cased English with punctuation" directs the model to produce formatted text directly, effectively performing ASR and ITN in a single pass.
Experimental Setup and Results
The model was evaluated using several large-scale datasets, including Libriheavy (5,000 hours of narrated audiobooks with original casing/punctuation), NPR (2,000 hours of diverse conversational recordings), and LibriSpeech (for word-level biasing tests).
Key Performance Metrics
- Significant Error Reduction: On the Libriheavy dataset, PromptASR achieved a 21.9% relative reduction in Word Error Rate (WER) compared to a non-prompted baseline. On the NPR dataset, it achieved a 6.8% reduction.
- Contextual Sensitivity: The model's performance scales with the amount of context provided. As shown in the experimental results, increasing the number of history utterances used as content prompts consistently lowers the WER.
- Rare Word Accuracy: In word-level biasing tasks, the model demonstrated high selectivity, achieving a 13.4% relative WER reduction even when the prompt list contained up to 1,000 "distractor" words not present in the audio.
- Style Controllability: The system successfully toggled between different output formats (e.g., all-caps vs. mixed-case with punctuation) based solely on the provided style prompt, without compromising recognition accuracy.
Conclusion
PromptASR represents a significant step toward more flexible and context-aware speech recognition. By unifying semantic context, keyword biasing, and stylistic control into a single cross-attention mechanism, the framework reduces the need for complex multi-stage pipelines. While the addition of a text encoder introduces some computational overhead, the gains in accuracy and versatility provide a strong foundation for future integration with even larger language models.
Summary updated