Identifying symptom etiologies using syntactic patterns and large language models

Hillel Taub-Tabib, Yosi Shamay, Micah Shlain, Menny Pinhasov, Mark J. Polak, Aryeh Tiktinsky, Sigal Rahamimov, Dan Bareket, Ben Eyal, Moriya Kassis, Yoav Goldberg, Tal Kaminski Rosenberg, Simon Vulfsons, Maayan Ben SassonScientific 2024 · DOI

Abstract

Differential diagnosis is a crucial aspect of medical practice, as it guides clinicians to accurate diagnoses and effective treatment plans. Traditional resources, such as medical books and services like UpToDate, are constrained by manual curation, potentially missing out on novel or less common findings. This paper introduces and analyzes two novel methods to mine etiologies from scientific literature. The first method employs a traditional Natural Language Processing (NLP) approach based on syntactic patterns. By using a novel application of human-guided pattern bootstrapping patterns are derived quickly, and symptom etiologies are extracted with significant coverage. The second method utilizes generative models, specifically GPT-4, coupled with a fact verification pipeline, marking a pioneering application of generative techniques in etiology extraction. Analyzing this second method shows that while it is highly precise, it offers lesser coverage compared to the syntactic approach. Importantly, combining both methodologies yields synergistic outcomes, enhancing the depth and reliability of etiology mining.

Introduction

In clinical practice, a "differential diagnosis" involves weighing all possible causes for a patient's symptoms. Currently, clinicians rely heavily on curated resources like UpToDate or medical textbooks. However, these resources suffer from a "manual curation bottleneck": they are slow to update and often miss rare or newly discovered etiologies. To address this, we explore automated text mining across 25 million PubMed abstracts to create a scalable, highly accurate way to extract symptom-cause relationships.

Methodology

We propose two distinct but synergistic methods for etiology extraction:

1. Human-in-the-Loop Syntactic Bootstrapping

Using the SPIKE extractive search system, we employed a bootstrapping method. A medical expert begins with a single "seed tuple" (e.g., sciatica is caused by herniation). The system identifies similar syntactic structures in PubMed and suggests new patterns. A human expert then vets these patterns—a process taking approximately four hours—resulting in 60 high-quality patterns applicable across the entire corpus.

Overview of the human-in-the-loop syntactic pattern bootstrapping process.

2. LLM Generation with Fact Verification

To leverage generative models while mitigating "hallucinations," we developed a multi-stage verification pipeline for GPT-4. The process involves:

  • Item Generation: GPT-4 generates a list of potential etiologies for a symptom.
  • Evidence Retrieval: Finding the top 50 sentences in PubMed linking the symptom and cause using dense embeddings.
  • Ranking & Selection: Using an LLM to select the three most convincing sentences.
  • Vetting: A final "Yes/No" check to ensure the evidence supports the causal claim.

Finally, results were normalized using BioSyn and the MEDIC ontology to ensure synonymous terms (e.g., "LDH" and "lumbar disc herniation") were unified.

Results and Key Insights

Performance and the "Long Tail"

Our evaluation against reference standards (UpToDate and textbooks) for symptoms like hiccups, jaundice, and chest pain revealed a clear trade-off. GPT-4 with the verification pipeline achieved 100% precision on primary test symptoms but struggled with the "long tail" of rare medical knowledge, reaching only 23.45% recall. Conversely, syntactic patterns achieved a significantly higher macro-averaged recall of 68.17%, proving far superior at capturing rare cases not stored in the LLM's internal weights.

Synergistic Complementarity

A striking finding was the minimal overlap between the two methods. GPT-4 identified causes that syntactic patterns missed, and vice-versa. Combining both methods increased recall to approximately 85%, significantly outperforming any single source, including the "gold standard" curated references.

Venn diagrams illustrating the overlap in etiologies found by syntactic patterns, GPT-4, and reference standards.

Conclusion

Automated methods can identify many valid etiologies absent from traditional curated sources. While LLMs provide high precision for common knowledge, syntactic patterns are essential for capturing the breadth of medical literature. Future work will expand this mining to full-text articles and explore Retrieval-Augmented Generation (RAG) to further enhance LLM performance on rare medical facts.

Summary updated

Abstract

Differential diagnosis is a crucial aspect of medical practice, as it guides clinicians to accurate diagnoses and effective treatment plans. Traditional resources, such as medical books and services like UpToDate, are constrained by manual curation, potentially missing out on novel or less common findings. This paper introduces and analyzes two novel methods to mine etiologies from scientific literature. The first method employs a traditional Natural Language Processing (NLP) approach based on syntactic patterns. By using a novel application of human-guided pattern bootstrapping patterns are derived quickly, and symptom etiologies are extracted with significant coverage. The second method utilizes generative models, specifically GPT-4, coupled with a fact verification pipeline, marking a pioneering application of generative techniques in etiology extraction. Analyzing this second method shows that while it is highly precise, it offers lesser coverage compared to the syntactic approach. Importantly, combining both methodologies yields synergistic outcomes, enhancing the depth and reliability of etiology mining.
BibTeX
@inproceedings{taubtabib2024identifying,
  author = {Hillel Taub-Tabib and Yosi Shamay and Micah Shlain and Menny Pinhasov and Mark J. Polak and Aryeh Tiktinsky and Sigal Rahamimov and Dan Bareket and Ben Eyal and Moriya Kassis and Yoav Goldberg and Tal Kaminski Rosenberg and Simon Vulfsons and Maayan Ben Sasson},
  title = {{Identifying symptom etiologies using syntactic patterns and large language models}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-65645-6},
}
Figure