Introduction
Automatic Speech Recognition (ASR) systems have reached impressive levels of accuracy in general domains, yet they continue to struggle with the "rare word" problem. Specialized jargon, technical terminology, and unique personal names often fall outside the high-frequency distribution of training data, leading to significant transcription errors in professional and academic settings.
In contexts such as online conferences, university lectures, or corporate meetings, these rare words are frequently visible on presentation slides. While "contextual biasing"—the practice of providing the model with a list of potential keywords—is a known technique to mitigate this, most existing methods treat these words as an unordered "bag of words." This approach ignores the rich, long-form relationships and sequential structure present in slide text. LCB-net (Long-Context Biasing network) is designed to bridge this gap by treating slide text as a synchronized, long-context source that informs the audio transcription process in real-time.
Methodology
LCB-net introduces a sophisticated bi-encoder architecture and specialized attention mechanisms to integrate visual and textual context into the ASR pipeline.
Bi-Encoder and Dual Cross-Attention
Unlike traditional models that rely on a single encoder, LCB-net utilizes two distinct encoders: one for processing audio signals and another for processing "long-context" text extracted from slides via Optical Character Recognition (OCR). To ensure these two streams interact effectively, the framework employs Dual Cross-Attention:
- Audio-Context (AC) Attention: This mechanism injects relevant information from the slides directly into the audio features, allowing the model to "listen" for terms it has "seen."
- Context-Audio (CA) Attention: This allows the text encoder to understand which specific portions of the slide are currently being discussed, helping the model maintain temporal alignment between the speech and the visual aids.
Biasing Prediction Module
A core innovation of LCB-net is the Biasing Prediction Module. This component explicitly predicts which phrases from the provided context are present in the audio stream. By utilizing 1D-convolutions to capture local patterns (such as sub-word units) and a Binary Cross Entropy (BCE) loss function, the module sharpens the model's focus on relevant terms. This explicit supervision encourages the model to rely more heavily on the provided context when it matches the acoustic signal.
BPE-based Simulation
To improve robustness, the researchers implemented a simulation strategy during training based on Byte Pair Encoding (BPE). Rather than biasing the model toward whole words only, the system simulates biased phrases using sub-word units. This makes the model more resilient to variations in word length and morphological changes, ensuring that a partial match (e.g., a prefix) can still help the model correctly identify a complex technical term.
Experimental Evaluation
The researchers evaluated LCB-net across two primary datasets to test both its specialized utility and its generalizability:
- SlideSpeech: A large-scale audio-visual corpus containing over 1,000 hours of video with synchronized slides. This served as the primary benchmark for assessing how well the model utilizes long-context visual information.
- LibriSpeech: A standard ASR benchmark consisting of 960 hours of audiobook readings. This was used to ensure the model maintains high performance even in domains without synchronized visual slides.
Key Results
LCB-net demonstrated significant improvements over baseline ASR models and previous state-of-the-art biasing methods:
- SlideSpeech Performance: The model achieved a relative reduction in Word Error Rate (WER) of 9.4%. Crucially, it reduced the Biased-WER (B-WER)—the error rate specifically for rare and technical terms—by 10.9%.
- LibriSpeech Performance: The gains were even more pronounced on the LibriSpeech benchmark, with a 23.8% relative WER reduction and a 35.4% reduction in B-WER compared to the baseline.
- System Robustness: Testing revealed that LCB-net does not become overly dependent on the presence of slides; it maintains high performance even when the biasing list is empty, indicating that the architecture does not degrade standard ASR capabilities.
Insights and Future Directions
The success of LCB-net highlights that context is most effective when treated as a sequence rather than a list. By using a transformer-based context encoder, the model learns long-range dependencies between related technical terms on a slide. Furthermore, the use of explicit supervision through the BCE loss proves that guiding the model to actively predict biased phrases leads to much more confident and accurate attention maps.
Despite these gains, the system's performance remains tied to the quality of the OCR. Future work will focus on making the model more resilient to low-quality video or unreadable text. Additionally, the researchers aim to integrate the keyword extraction step more tightly into the end-to-end process and scale the model for industrial-level production environments.
Summary updated