Abstract
Extracting phenotypes from clinical text has been shown to be useful for a variety of clinical use cases such as identifying patients with rare diseases. However, reasoning with numerical values remains challenging for phenotyping in clinical text, for example, temperature 102F representing Fever. Current state-of-the-art phenotyping models are able to detect general phenotypes, but perform poorly when they detect phenotypes requiring numerical reasoning. We present a novel unsupervised methodology leveraging external knowledge and contextualized word embeddings from ClinicalBERT for numerical reasoning in a variety of phenotypic contexts. Comparing against unsupervised benchmarks, it shows a substantial performance improvement with absolute gains on generalized Recall and F1 scores up to 79% and 71%, respectively. In the supervised setting, it also surpasses the performance of alternative approaches with absolute gains on generalized Recall and F1 scores up to 70% and 44%, respectively.
Introduction
Extracting phenotypes—observable clinical characteristics—from electronic health records (EHRs) is vital for identifying rare diseases and predicting patient outcomes. However, current state-of-the-art Natural Language Processing (NLP) models have a "numerical blind spot." While a model might recognize the word "Fever," it often fails to understand that "temperature of 102F" implies the same condition. Clinical text is often unstructured and dense, with numbers appearing in clusters (e.g., "BP 120/80, HR 70"), units frequently omitted, and context-dependent interpretations. Most existing tools either ignore numbers entirely or require massive amounts of expensive, human-labeled data to learn these relationships.
Methods
The researchers proposed an unsupervised four-step pipeline that integrates external medical knowledge with contextualized word embeddings to perform numerical reasoning.
1. Knowledge Base Creation
A reference table was constructed mapping 33 common numeric entities (e.g., heart rate, serum creatinine) to their normal reference ranges and corresponding Human Phenotype Ontology (HPO) codes for "high," "low," and "normal" states. This external knowledge allows the system to interpret numerical values deterministically.
2. Extraction
Using regular expressions and the Stanza NLP library, the system extracts numbers and "lexical candidates"—nearby nouns, verbs, or adjectives such as "pyrexia" or "increased"—from the clinical text.
3. Contextualized Embeddings
The system utilizes ClinicalBERT, fine-tuned with a Semantic Textual Similarity (STS) objective. This ensures that words representing similar clinical concepts (e.g., "pyrexia" and "temperature") are positioned closely in the embedding space, facilitating accurate entity matching.
4. Reasoning and Assignment
The system calculates the cosine similarity between extracted lexical candidates and known numeric entities. Once a match is established (e.g., "pyrexia" matching "temperature"), the associated numerical value (e.g., 102F) is compared against the knowledge base's reference ranges to assign the appropriate phenotype (e.g., Fever). To handle missing units, the model evaluates ratios against different scales (Celsius vs. Fahrenheit) to select the most realistic interpretation.
Results and Key Insights
The model was evaluated using the MIMIC-III database and a curated test set of 705 EHR notes containing over 20,000 gold-standard phenotype annotations, specifically focusing on 1,121 "Numerical Reasoning" (NR) specific phenotypes.
- Superiority over Unsupervised Baselines: Traditional unsupervised tools like NCBO or NCR scored 0% on numerical reasoning tasks as they lack numerical processing capabilities. The proposed model achieved a 79% generalized Recall and a 71% F1 score.
- Outperforming Supervised Models: Surprisingly, the unsupervised model outperformed supervised baselines, which often suffer from "frequency bias"—over-predicting common conditions regardless of numerical evidence. The NR model showed absolute gains of up to 70% in Recall and 44% in F1 score.
- Impact of STS Fine-tuning: Ablation studies showed that using fine-tuned embeddings instead of simple keyword matching improved Recall from 26.4% to 68.8%, highlighting the importance of contextual understanding.
- Zero-Shot Capability: By relying on a knowledge base rather than specific training examples, the model can be easily extended to new clinical measurements by simply updating the reference table.
Discussion and Limitations
While the framework shows significant promise, certain challenges remain. The model can be misled by complex contextual negation or supplemental information (e.g., oxygen saturation levels interpreted differently when a patient is on supplemental oxygen). Additionally, the current knowledge base is static and manually curated; future iterations could automate its expansion from medical literature. Finally, expanding from sentence-level to document-level reasoning would allow the system to capture clinical trends over time.
Summary updated