Abstract
Differential diagnosis is a crucial aspect of medical practice, as it guides clinicians to accurate diagnoses and effective treatment plans. Traditional resources, such as medical books and services like UpToDate, are constrained by manual curation, potentially missing out on novel or less common findings. This paper introduces and analyzes two novel methods to mine etiologies from scientific literature. The first method employs a traditional Natural Language Processing (NLP) approach based on syntactic patterns. By using a novel application of human-guided pattern bootstrapping patterns are derived quickly, and symptom etiologies are extracted with significant coverage. The second method utilizes generative models, specifically GPT-4, coupled with a fact verification pipeline, marking a pioneering application of generative techniques in etiology extraction. Analyzing this second method shows that while it is highly precise, it offers lesser coverage compared to the syntactic approach. Importantly, combining both methodologies yields synergistic outcomes, enhancing the depth and reliability of etiology mining.
Introduction
In clinical practice, a "differential diagnosis" involves weighing all possible causes for a patient's symptoms. Currently, clinicians rely heavily on curated resources like UpToDate or medical textbooks. However, these resources suffer from a "manual curation bottleneck": they are slow to update and often miss rare or newly discovered etiologies. To address this, we explore automated text mining across 25 million PubMed abstracts to create a scalable, highly accurate way to extract symptom-cause relationships.
Methodology
We propose two distinct but synergistic methods for etiology extraction:
1. Human-in-the-Loop Syntactic Bootstrapping
Using the SPIKE extractive search system, we employed a bootstrapping method. A medical expert begins with a single "seed tuple" (e.g., sciatica is caused by herniation). The system identifies similar syntactic structures in PubMed and suggests new patterns. A human expert then vets these patterns—a process taking approximately four hours—resulting in 60 high-quality patterns applicable across the entire corpus.
2. LLM Generation with Fact Verification
To leverage generative models while mitigating "hallucinations," we developed a multi-stage verification pipeline for GPT-4. The process involves:
- Item Generation: GPT-4 generates a list of potential etiologies for a symptom.
- Evidence Retrieval: Finding the top 50 sentences in PubMed linking the symptom and cause using dense embeddings.
- Ranking & Selection: Using an LLM to select the three most convincing sentences.
- Vetting: A final "Yes/No" check to ensure the evidence supports the causal claim.
Finally, results were normalized using BioSyn and the MEDIC ontology to ensure synonymous terms (e.g., "LDH" and "lumbar disc herniation") were unified.
Results and Key Insights
Performance and the "Long Tail"
Our evaluation against reference standards (UpToDate and textbooks) for symptoms like hiccups, jaundice, and chest pain revealed a clear trade-off. GPT-4 with the verification pipeline achieved 100% precision on primary test symptoms but struggled with the "long tail" of rare medical knowledge, reaching only 23.45% recall. Conversely, syntactic patterns achieved a significantly higher macro-averaged recall of 68.17%, proving far superior at capturing rare cases not stored in the LLM's internal weights.
Synergistic Complementarity
A striking finding was the minimal overlap between the two methods. GPT-4 identified causes that syntactic patterns missed, and vice-versa. Combining both methods increased recall to approximately 85%, significantly outperforming any single source, including the "gold standard" curated references.
Conclusion
Automated methods can identify many valid etiologies absent from traditional curated sources. While LLMs provide high precision for common knowledge, syntactic patterns are essential for capturing the breadth of medical literature. Future work will expand this mining to full-text articles and explore Retrieval-Augmented Generation (RAG) to further enhance LLM performance on rare medical facts.
Summary updated