Abstract
This paper presents an ontology-driven treatment article retrieval system developed and experimented using the data and ground truths provided by the TREC 2017 precision medicine track. The key aspects of our system include: meaningful integration of various disease, gene, and drug name ontologies, training of a novel perceptron model for article relevance labeling, a ranking module that considers additional factors such as journal impact and article publication year, and comprehensive query matching rules. Experimental results demonstrate that our proposed system considerably outperforms the results of the best participating system of the TREC 2017 precision medicine challenge.
Introduction
In precision oncology, clinicians must match a patient’s specific genetic profile and disease type to the latest targeted therapies. However, the sheer volume of biomedical literature—over 30 million papers on PubMed alone—makes manual searching nearly impossible. The challenge isn't just finding "relevant" keywords; it's dealing with the high variability of medical language. A single gene or disease might be referred to by many synonyms across different papers. Furthermore, clinicians need the most recent, high-quality evidence from reputable journals to make life-critical decisions. This paper addresses the need for an automated system that understands medical context and prioritizes high-impact research.
Methods
The researchers developed an end-to-end pipeline consisting of four main stages, as illustrated in the system architecture (Figure 1).
- Ontology-Driven Expansion: The system uses a "unified ontology" (merging UMLS, MeSH, and ICD-10) for disease synonyms. It also utilizes NCBI and GeneMania for gene synonyms, OncoKB for genetic variants, and RxNorm for drug names.
- Heuristic Query Formulation: Complex Elasticsearch queries are generated with strict rules. Articles must contain the disease, gene, and either a variant or treatment keyword, while respecting patient demographics (e.g., filtering out male-specific trials for female patients).
- Neural Relevance Labeling: A Perceptron model filters noise using frequency features (bag-of-words and phrases) of disease and drug names in titles and abstracts to classify paper relevance.
- Two-Stage Ranking: Primary scoring uses Elasticsearch’s hybrid relevance (tf-idf and coordinate matching). A secondary score prioritizes publication recency and Journal Impact Factor (using Google Scholar’s H5 index) to break ties among papers with similar relevance.
Results and Discussion
The system was evaluated using the TREC 2017 Precision Medicine track dataset, which includes 30 synthetic patient profiles and over 21,000 manually labeled PubMed articles.
As shown in Figure 3, the best configuration (OncoKB + Ranking + Perceptron) achieved a Precision at 5 (P5) of 0.7800. This significantly outperformed the top system in the TREC 2017 challenge, which had a P5 of 0.7000—representing an 11.4% improvement. Ablation studies highlighted the importance of specialized knowledge; removing the OncoKB ontology caused the P5 to drop to 0.7533, proving that specialized genetic knowledge bases are vital for precision medicine tasks.
Key Insights
- Ontologies as Force Multipliers: Integration of OncoKB is critical for recall because genetic variants are described with high variability (e.g., "G12C" vs "Gly12Cys").
- Quality-Aware Ranking: Prioritizing recent, high-impact journal articles provides more clinical value than keyword matching alone.
- Effective Simple Models: A well-tuned Perceptron using frequency-based features can effectively refine keyword-based search results, showing that complex architectures aren't always necessary for effective filtering.
Conclusion
While the system demonstrates superior performance, challenges remain in maintaining unified ontologies as medical terminology evolves. Future work could explore more advanced neural architectures, such as BERT-based models, to capture deeper semantic nuances beyond frequency-based features, and address dependencies on indexing codes like MeSH for real-time article processing.
Summary updated