Abstract
Standard language models often struggle with "rare" words, frequently omitting them in favor of more common terms. In healthcare, these rare words—such as specific medications or symptoms—are critical. This paper introduces a "utilization rate" regularization technique that leverages external medical knowledge to ensure vital concepts are preserved when transforming doctor-patient dialogues into official medical instructions.
Introduction
When a doctor and patient converse, they frequently mention specific, rare terms: a niche medication like warfarin or a specific condition like serotonin syndrome. When AI is employed to summarize these conversations into "after-visit care instructions," standard sequence-to-sequence (seq2seq) models tend to ignore these rare tokens. This is due to a fundamental bias in machine learning: models are trained to maximize the probability of the next word based on frequency. If a word is rare in the training data, the model "learns" that it is unlikely to be the correct choice, even if it appeared clearly in the source conversation.
The Problem: Rare Word Omission in Clinical Settings
While "copy mechanisms" exist to help models repeat words from the input, they often fail to distinguish between a rare but important medical term and a rare but irrelevant filler word. This leads to "hallucinations" or omissions that can be dangerous in a clinical setting.
Methodology: Utilization Rate Regularization
The authors propose a method to "uplift" the probability of important rare tokens by injecting external knowledge into the training process.
High Utilization Concepts (HUCs)
The researchers define a category of concepts that are rare overall but have a high "utilization rate" — meaning if they appear in the source (the dialogue), they almost always appear in the target (the care instructions). To calculate this rate for extremely rare words where data is sparse, they use external medical ontologies (UMLS, SNOMED-CT, and RXNorm). They group rare words into "equivalence classes" (e.g., grouping specific drug names under the semantic type "Medication"). This allows them to calculate a stable utilization rate for the category, which they then apply to the individual rare words.
Knowledge-Informed Loss
They add a new term to the model's loss function during training. This "utilization loss" penalizes the model if it underestimates the marginal probability of these high-utilization concepts. The most successful approach was "Semantic Weighted," which weights concepts based on the utilization rate of their broader medical category.
Experimental Setup and Datasets
The study used a custom dataset of 14,000 medical encounters from a virtual primary care platform. The source consisted of text-based conversations between doctors and patients, while the reference was the actual care plans written by the doctors in the Electronic Health Record (EHR). The authors used FastText embeddings and cosine similarity to create a parallel corpus of 48,000 source-reference pairs.
Results and Discussion
The "Semantic Weighted" model was the clear winner across both automated and human metrics. It achieved a Concept-F1 score of ~75.0, a massive jump from the baseline's 57.4. In human evaluations, five medical doctors rated the model's relevance at 3.78/5, compared to the baseline's 2.50/5. Clinical usability was rated at 3.99/5, significantly higher than the baseline (3.18) or the Dynamic Beam Allocation (DBA) approach (3.35).
A key insight was the "100x Gap": for many medical concepts, the utilization rate is 10 to 100 times higher than their marginal probability in general text. By analyzing the model's entropy, the researchers found that their regularization significantly reduced uncertainty at the beginning of the generation process, helping the model "commit" to the correct medical concepts early.
Conclusion
While the method shows great promise, it currently depends on high-quality external knowledge sources like UMLS. Future work will explore scalability to massive Large Language Models (LLMs) and the incorporation of differential privacy mechanisms to mitigate the risk of "inference leakage" associated with rare medical conditions.
Summary updated