Combining Language Models For Specialized Domains: A Colorful Approach

Daniel Eitan, Menachem Pirchi, Neta Glazer, Shai Meital, Gil Ayache, Gidon Krendel, Aviv Shamsian, Aviv Navon, Gill Hetz, Joseph KeshetarXiv 2023 · OpenReview

Abstract

General purpose language models (LMs) encounter difficulties when processing domain-specific jargon and terminology, which are frequently utilized in specialized fields such as medicine or industrial settings. Moreover, they often find it challenging to interpret mixed speech that blends general language with specialized jargon. This poses a challenge for automatic speech recognition systems operating within these specific domains. In this work, we introduce a novel approach that integrates domain-specific or secondary LM into general-purpose LM. This strategy involves labeling, or "coloring", each word to indicate its association with either the general or the domain-specific LM. We develop an optimized algorithm that enhances the beam search algorithm to effectively handle inferences involving colored words. Our evaluations indicate that this approach is highly effective in integrating jargon into language tasks. Notably, our method substantially lowers the error rate for domain-specific words without compromising performance in the general domain.

Introduction

Standard Large Language Models (LMs) excel at general conversation but frequently struggle with "mixed speech"—sentences that blend everyday language with highly specific technical terminology (jargon). This is particularly problematic in Automatic Speech Recognition (ASR) for private industrial or medical environments where specialized terms are rare in public training data.

Traditional solutions typically rely on interpolation, which averages the predictions of a general LM and a jargon-specific LM. However, interpolation suffers from several flaws: it often uses static weights that do not adapt to sentence context, it can dilute probability scores leading to errors, and it tends to "wash out" rare jargon terms through over-smoothing.

Methodology: Word Coloring

To address these limitations, we propose a novel decoding strategy termed "Word Coloring." Instead of averaging model outputs, we treat the source of each word (General vs. Jargon) as a hidden variable. Each word in a sequence is assigned a "color" $c$ representing its source lexicon.

Comparison of traditional interpolation methods (A, B) versus the proposed word coloring approach (C).

The system jointly optimizes for both the word and its source, allowing the model to dynamically "jump" between the general and jargon LMs on a word-by-word basis. This ensures the model utilizes the lexicon that provides the most likely fit for the acoustic signal.

We adapted the standard Connectionist Temporal Classification (CTC) beam search algorithm to support this. In our implementation, while the first letter of a word can originate from any available lexicon, once a "color" is assigned to the start of a word, the algorithm forces subsequent letters to remain within that same lexicon to maintain internal consistency. This approach is computationally efficient and scales easily to multiple specialized lexicons.

Experimental Results

We evaluated the "Colorful Approach" across four distinct, real-world datasets: Industrial English (noisy mechanical jargon), Industrial Thai, and two medical datasets (respiratory cases and doctor-patient interviews).

Quantitative results showing Character Error Rate (CER) and Word Error Rate (WER) across four specialized datasets.

As shown in the results table, our method consistently outperformed traditional interpolation baselines:

  • Industrial English: The most significant improvement was observed here, with the WER dropping to 11.9%, compared to 36.8% for a general LM and 15.9% for the best interpolation baseline.
  • Medical Domains: The method achieved the lowest WER in both medical datasets (12.3% and 11.4%), successfully identifying complex drug names like clozaril.
  • General Performance: Crucially, the improvement in jargon recognition did not come at the cost of general language accuracy, avoiding the common "general penalty" associated with narrow fine-tuning.

Discussion and Conclusion

The "aha moment" of this research is the realization that interpolation is often a compromise that satisfies neither model. By allowing the system to pick a "winner" for each word through dynamic switching, we preserve the high-confidence peaks of specialized models. Furthermore, assigning a source to each word provides valuable metadata for downstream tasks like automated medical coding.

Future work will explore learning transition probabilities between colors rather than assuming a uniform distribution, and expanding this logic to tasks like Machine Translation where domain-switching is equally prevalent.

Summary updated

Abstract

General purpose language models (LMs) encounter difficulties when processing domain-specific jargon and terminology, which are frequently utilized in specialized fields such as medicine or industrial settings. Moreover, they often find it challenging to interpret mixed speech that blends general language with specialized jargon. This poses a challenge for automatic speech recognition systems operating within these specific domains. In this work, we introduce a novel approach that integrates domain-specific or secondary LM into general-purpose LM. This strategy involves labeling, or "coloring", each word to indicate its association with either the general or the domain-specific LM. We develop an optimized algorithm that enhances the beam search algorithm to effectively handle inferences involving colored words. Our evaluations indicate that this approach is highly effective in integrating jargon into language tasks. Notably, our method substantially lowers the error rate for domain-specific words without compromising performance in the general domain.
BibTeX
@misc{eitan2023combining,
  author = {Daniel Eitan and Menachem Pirchi and Neta Glazer and Shai Meital and Gil Ayache and Gidon Krendel and Aviv Shamsian and Aviv Navon and Gill Hetz and Joseph Keshet},
  title = {{Combining Language Models For Specialized Domains: A Colorful Approach}},
  year = {2023},
  eprint = {2310.19708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.19708},
}
Figure