TL;DR
This paper introduces a novel, scalable pipeline for automatically discovering semantic associations between medical concepts, specifically focusing on disease-symptom pairs. By combining traditional association rule mining with advanced NLP techniques and leveraging the vastness of PubMed abstracts, the system efficiently surfaces potential relationships, offering a valuable tool for accelerating medical discovery and supporting clinical decision-making.
What Problem Does This Paper Address?
The sheer volume of medical literature makes it incredibly challenging for researchers and clinicians to stay abreast of new discoveries and identify subtle connections between diseases and their symptoms. Existing methods for extracting these relationships often rely on manual annotation, which is time-consuming and expensive, or they fail to fully leverage the semantic richness of the text. This leads to a lag in updating knowledge bases and a missed opportunity to accelerate the identification of new symptoms, disease patterns, and potential diagnostic tools, especially crucial for public health surveillance and preventative care.
Methods
The proposed pipeline is an end-to-end system designed for scalability and ease of updating. It consists of three main components:
- Document Annotation and Indexing: Incoming medical literature (PubMed abstracts) is first split into sentences. Each sentence is then annotated using Scispacy, a biomedical NLP library, to identify entities (like diseases and symptoms) and their UMLS semantic types. These annotated sentences, along with entity identifiers, are stored in an Elasticsearch index. This allows for efficient retrieval of sentences containing specific diseases and their co-occurring entities.
- Association Rules Mining (ARM): This component uses a frequency-based approach to discover potential disease-symptom associations. It queries the Elasticsearch index to find sentences mentioning a given disease and then identifies co-occurring symptoms. It calculates the "support" (frequency of co-occurrence) and "confidence" of association rules (e.g., Disease ⇒ Symptom).
- UMLS Semantic Network Based Pruning & Binary Relation Classification (BRC):
- UMLS Pruning: To filter out irrelevant associations (e.g., a disease and a body part), the system leverages the UMLS Semantic Network to ensure that identified entities are semantically related to the "disease" and "symptom" categories.
- BRC Model: A crucial step involves a binary relation classification model, fine-tuned on a BERT transformer (PubMedBERT) trained on medical text. This model is trained using distant supervision to predict whether two entities in a sentence are semantically related. It takes sentences with tagged entities and outputs a probability score. The final score for a disease-symptom pair is a weighted combination of the ARM and BRC scores.
The system also incorporates a human-in-the-loop verification mechanism for interpreting and validating discovered associations.
Datasets
- PubMed Abstracts: The system was trained and evaluated on a collection of over 14 million PubMed abstracts published between 2000 and 2022.
- Disease-Symptoms Knowledge Database (Wang et al., 2008): This online database, published by Columbia University, served as the ground truth for evaluating the system's performance. The authors note that they used a subset of this data, focusing on 54 diseases for which symptoms were present in their PubMed abstract collection. They also manually expanded this ground truth with new associations identified by their system.
Key Insights and Ideas
- Semantic Richness Matters: Simply counting co-occurrences (ARM) is useful but insufficient. Incorporating a semantic relation classifier (BRC) trained on deep learning models significantly improves the accuracy and relevance of discovered associations.
- Leveraging UMLS for Filtering: The UMLS Semantic Network acts as a vital intermediary, helping to prune spurious associations and ensure that the identified relationships are semantically plausible within the medical domain.
- Scalability and Updatability: The use of Elasticsearch and an automated annotation pipeline allows the system to efficiently ingest and process millions of documents, making it easily updatable with new research.
- Interpretability through Evidence: The system doesn't just provide a list of associations; it also returns supporting sentences from the literature, allowing users to understand why an association is suggested and to verify its validity.
Key Results
The paper reports strong performance in discovering disease-symptom associations:
- Recall of 1.00: The proposed pipeline, combining ARM and BRC, successfully retrieved all relevant disease-symptom pairs from the ground truth data within the top 10 ranked symptoms.
- Improved Precision: The combined ARM and BRC approach achieved a P@1 score of 0.57, P@3 of 0.33, and P@5 of 0.25, outperforming individual ARM or BRC models.
- Qualitative Validation: Manual evaluation and expansion of the ground truth dataset added 70 new disease-symptom pairs, demonstrating the system's ability to uncover novel or previously uncatalogued relationships.
- High F1 Score for BRC: The binary relation classification model achieved an F1 score of 0.94 on its validation set, indicating high accuracy in identifying semantically related entity pairs within sentences.
Limitations and Future Work
- Annotation Accuracy: The accuracy of the system is inherently limited by the quality of the entity annotator. Misrecognized acronyms or entities can lead to inaccurate associations.
- UMLS Granularity: The available UMLS semantic types might not always capture the precise nuances of medical relationships, potentially introducing noise.
- PubMed Abstract Focus: The system currently relies only on PubMed abstracts. Future work could explore incorporating full-text articles, which may contain richer information.
- Veracity Over Time: The paper acknowledges the need to explore how to handle situations where scientific opinions or findings change over time.
- Learning to Rank: The authors plan to investigate machine learning-based approaches for combining ARM and BRC scores more effectively.
- Ethical Considerations: The authors emphasize that the system is a tool to support medical professionals, not replace them. They highlight the importance of user awareness regarding potential model errors and the need for safeguards in clinical applications. They also note the importance of unbiased data collections to avoid performance disparities across different populations.
Summary updated