Abstract
General purpose language models (LMs) encounter difficulties when processing domain-specific jargon and terminology, which are frequently utilized in specialized fields such as medicine or industrial settings. Moreover, they often find it challenging to interpret mixed speech that blends general language with specialized jargon. This poses a challenge for automatic speech recognition systems operating within these specific domains. In this work, we introduce a novel approach that integrates domain-specific or secondary LM into general-purpose LM. This strategy involves labeling, or "coloring", each word to indicate its association with either the general or the domain-specific LM. We develop an optimized algorithm that enhances the beam search algorithm to effectively handle inferences involving colored words. Our evaluations indicate that this approach is highly effective in integrating jargon into language tasks. Notably, our method substantially lowers the error rate for domain-specific words without compromising performance in the general domain.
Introduction
Standard Large Language Models (LMs) excel at general conversation but frequently struggle with "mixed speech"—sentences that blend everyday language with highly specific technical terminology (jargon). This is particularly problematic in Automatic Speech Recognition (ASR) for private industrial or medical environments where specialized terms are rare in public training data.
Traditional solutions typically rely on interpolation, which averages the predictions of a general LM and a jargon-specific LM. However, interpolation suffers from several flaws: it often uses static weights that do not adapt to sentence context, it can dilute probability scores leading to errors, and it tends to "wash out" rare jargon terms through over-smoothing.
Methodology: Word Coloring
To address these limitations, we propose a novel decoding strategy termed "Word Coloring." Instead of averaging model outputs, we treat the source of each word (General vs. Jargon) as a hidden variable. Each word in a sequence is assigned a "color" $c$ representing its source lexicon.
Dynamic Switching and Beam Search
The system jointly optimizes for both the word and its source, allowing the model to dynamically "jump" between the general and jargon LMs on a word-by-word basis. This ensures the model utilizes the lexicon that provides the most likely fit for the acoustic signal.
We adapted the standard Connectionist Temporal Classification (CTC) beam search algorithm to support this. In our implementation, while the first letter of a word can originate from any available lexicon, once a "color" is assigned to the start of a word, the algorithm forces subsequent letters to remain within that same lexicon to maintain internal consistency. This approach is computationally efficient and scales easily to multiple specialized lexicons.
Experimental Results
We evaluated the "Colorful Approach" across four distinct, real-world datasets: Industrial English (noisy mechanical jargon), Industrial Thai, and two medical datasets (respiratory cases and doctor-patient interviews).
As shown in the results table, our method consistently outperformed traditional interpolation baselines:
- Industrial English: The most significant improvement was observed here, with the WER dropping to 11.9%, compared to 36.8% for a general LM and 15.9% for the best interpolation baseline.
- Medical Domains: The method achieved the lowest WER in both medical datasets (12.3% and 11.4%), successfully identifying complex drug names like clozaril.
- General Performance: Crucially, the improvement in jargon recognition did not come at the cost of general language accuracy, avoiding the common "general penalty" associated with narrow fine-tuning.
Discussion and Conclusion
The "aha moment" of this research is the realization that interpolation is often a compromise that satisfies neither model. By allowing the system to pick a "winner" for each word through dynamic switching, we preserve the high-confidence peaks of specialized models. Furthermore, assigning a source to each word provides valuable metadata for downstream tasks like automated medical coding.
Future work will explore learning transition probabilities between colors rather than assuming a uniform distribution, and expanding this logic to tasks like Machine Translation where domain-switching is equally prevalent.
Summary updated