TL;DR
This paper introduces RECAST, a novel method for improving Automatic Speech Recognition (ASR) by efficiently retrieving relevant keywords from large dictionaries without needing audio examples. RECAST leverages the internal states of a pre-trained ASR model to pinpoint keywords, significantly boosting accuracy and recall, especially for rare or domain-specific terms.
What Problem Does This Paper Address?
Recognizing specific, rare, or domain-specific terms in speech is crucial for many ASR applications (e.g., medical, financial). Traditional methods for "contextual biasing" these terms struggle to scale. Large keyword dictionaries lead to increased transcription errors and latency, making it impractical to include all possible keywords. Existing approaches either require audio examples for each keyword (which is infeasible for large sets) or rely on the ASR's own potentially flawed hypotheses. The core challenge is efficiently identifying a small, relevant subset of keywords for any given utterance from a massive list.
Methods
RECAST tackles this by repurposing the decoder states of a pre-trained encoder-decoder ASR model. The key innovation is a retrieval-augmented approach that doesn't require modifying the ASR backbone or using audio exemplars.
Here's a breakdown:
- Contrastive Retriever: A lightweight retrieval model is trained to align the ASR decoder's intermediate states (which implicitly encode audio-text relationships) with textual keyword embeddings. This is done using a contrastive loss, pushing matching pairs closer and non-matching pairs further apart in a shared embedding space.
- Keyword Indexing: Keywords are tokenized and their embeddings are stored in an efficient kNN index.
- Inference: During ASR decoding, at each step, the decoder's state is projected into an embedding and used to query the keyword index. This retrieves potential keyword matches.
- Span Aggregation and Ranking: Retrieved token-level matches are stitched together to form keyword hypotheses. These hypotheses are then ranked based on a combination of similarity, length, and positional diversity to select the top-K keywords.
- Contextual Prompting: The top-K retrieved keywords are then used to prompt a downstream ASR model (like Whisper or other LLMs), guiding it to produce more accurate transcriptions.
Datasets
- Training: LibriSpeech (960-hour corpus) was used to train the retrieval model.
- Evaluation:
- Entity-rich PRISM dataset: Contains out-of-domain utterances with named entities across various domains.
- Standard LibriSpeech benchmark: Used with curated keyword dictionaries from prior ASR studies.
- IndicVoices dataset: Used for evaluating performance on Hindi, a language with challenging tokenization characteristics.
Key Insights and Ideas
- Decoder states are rich sources of information: The paper demonstrates that the internal states of an ASR decoder, typically used for generating text, can be effectively repurposed as queries for retrieving relevant keywords. This bypasses the need for explicit audio-text alignment for keywords.
- Scalability without audio exemplars: RECAST offers a practical solution for large keyword dictionaries by avoiding the need for audio recordings of each keyword, a major bottleneck in prior methods.
- Language-agnostic potential: The approach is inherently language-agnostic, as it relies on text embeddings and ASR decoder states, making it applicable to languages where phonetic methods might fail (like Hindi).
- Importance of ranking: Simply retrieving keywords isn't enough; the ranking strategy (combining similarity, length, and positional diversity) is crucial for selecting the most relevant and impactful keywords.
Key Results
- Significant WER reduction: RECAST achieved up to a 41.3% overall WER improvement and a 54.3% relative reduction in entity WER on out-of-domain benchmarks.
- Improved recall: Achieved up to 2.5x higher recall in challenging settings, particularly for rare entities.
- Effective on Hindi: Demonstrated substantial improvements on Hindi, with recall increasing by up to 78.2% and E-WER dropping by 19.5% with specific settings.
- Low latency: Maintains a consistently low Real-Time Factor (RTF) of 0.013-0.015, even with keyword dictionaries scaling up to 4,000+ entities, making it suitable for real-time applications.
- Robustness across ASR models: While larger ASR backbones yield better results, RECAST shows competitive performance even with smaller models.
Limitations and Future Work
- Encoder-decoder dependency: The current implementation is tied to encoder-decoder ASR architectures. Extending it to CTC or RNN-T models would require new methods for alignment estimation.
- Benchmark limitations: Real-world deployments might involve tens of thousands of entities, for which suitable benchmarks are currently unavailable.
- Domain-specific fine-tuning: For specialized domains (e.g., medicine), further fine-tuning of the base ASR model could yield even better results, but this was not feasible in the current study.
- Two-stage pipeline: RECAST currently involves a separate retrieval step followed by ASR. Future work could explore tighter integration for a more streamlined pipeline.
- Potential for PII exposure and manipulation: The authors acknowledge risks if keyword dictionaries contain sensitive information or if malicious dictionaries are introduced.
- Multilingual extensions: While language-agnostic in principle, the current setup is language-specific. Future work could explore multilingual extensions with a shared retriever.
Summary updated