FindZebra: A search engine for rare diseases

Radu Dragusin, Paula Petcu, Christina Lioma, Birger Larsen, Henrik Jørgensen, Ingemar J. Cox, Lars Kai Hansen, Peter Ingwersen, Ole Winther

TL;DR

This paper introduces FindZebra, a specialized search engine designed to improve the accuracy of rare disease diagnosis for clinicians. By leveraging curated medical resources and a novel evaluation framework, FindZebra significantly outperforms general-purpose search engines like Google, demonstrating the value of domain-specific tools in complex diagnostic scenarios.

What Problem Does This Paper Address?

The internet has become an indispensable tool for medical professionals seeking information, particularly for rare or "orphan" diseases. These conditions are challenging due to their low prevalence, diverse symptoms, and scattered information. While clinicians often turn to popular web search engines (like Google) and databases (like PubMed) for diagnostic assistance, these tools are not specifically optimized for the nuances of rare disease diagnosis. This can lead to information overload, irrelevant results, and potentially missed diagnoses. The paper aims to evaluate the effectiveness of existing search tools for this specific task and to demonstrate the benefits of a specialized search engine.

Methods

The authors developed a comprehensive evaluation approach for rare disease diagnosis search. This involved:

  • A curated dataset: They compiled a high-quality dataset of rare disease information from authoritative sources, excluding less reliable content like blogs.
  • Diagnostic queries: 56 real-life, difficult clinical cases were used to create diagnostic queries, extracted directly from patient symptoms.
  • Evaluation metrics: Standard metrics like precision at rank k (P@k) and mean reciprocal rank (MRR) were used to quantitatively assess search engine performance.
  • FindZebra development: A specialized search engine, FindZebra, was built using open-source information retrieval technology (Indri). It utilizes the curated rare disease dataset and incorporates functionalities tailored for clinicians, including the use of UMLS medical concepts for clustering and ranking results.
  • Benchmarking: FindZebra was compared against Google Search, Google Custom Search (configured to emphasize the curated dataset or search the entire web), and PubMed.

Datasets

The primary dataset used for evaluation was a curated collection of 33,144 documents related to rare diseases, sourced from authoritative medical resources. This dataset was specifically compiled to exclude less reliable information. The queries were derived from 56 real-life diagnostic cases, with sources including clinician-generated queries, articles from the Orphanet Journal of Rare Diseases (OJRD), and the British Medical Journal (BMJ).

Key Insights and Ideas

  • General search engines are suboptimal for rare disease diagnosis: Despite their popularity, Google and PubMed struggle with the complexity and scattered nature of rare disease information, often returning irrelevant or noisy results.
  • Specialized search engines offer significant advantages: FindZebra, by focusing on a curated dataset and employing domain-specific functionalities, demonstrates a clear improvement in diagnostic accuracy and relevance.
  • The "algorithm dimension" matters: Even when restricting Google to the curated dataset (Google Restricted), its performance was inferior to FindZebra, suggesting that Google's general-purpose ranking algorithm is not ideal for this specialized task.
  • UMLS medical concepts enhance usability: FindZebra's ability to cluster and rank documents by UMLS medical concepts provides clinicians with a more structured and efficient way to navigate diagnostic information.
  • Query length and complexity are challenges: Longer, symptom-rich queries, common in rare disease diagnosis, pose difficulties for standard search engine algorithms.

Key Results

FindZebra significantly outperformed Google Search and PubMed across various metrics:

  • Mean Reciprocal Rank (MRR): FindZebra achieved an MRR of 0.385, compared to 0.206 for Google Search and lower scores for other variants. This indicates that the correct diagnosis appeared, on average, within the top 3 results for FindZebra.
  • Precision at Rank k (P@k): FindZebra showed higher precision, with 62.5% of queries having a relevant result in the top 10 and 67.9% in the top 20. Google Search achieved 28.6% and 32.1% respectively.
  • Unanswered Queries: FindZebra had significantly fewer unanswered queries (queries where no relevant result was found in the top 20) compared to other systems.
  • Performance on specific query sets: FindZebra showed particular strength on queries derived from OJRD and HLJ, which were longer and more complex.

Limitations and Future Work

  • Query bias: The authors acknowledge that the queries were based on descriptions of symptoms from case stories, and knowing the diagnosis might have inadvertently biased the symptom descriptions.
  • Limited index for some queries: FindZebra's index did not include documents for all diseases, leading to missed results for a few queries.
  • Short query limitations: For shorter queries (like those from BMJ), FindZebra's performance was comparable to Google Custom, suggesting that query expansion techniques could be beneficial.
  • Future work: The authors suggest expanding the dataset with more authoritative sources (e.g., UpToDate, Medline Plus) and exploring more advanced information retrieval models. A crucial future direction is the systematic collection of unbiased symptom-diagnosis association data, which would greatly improve evaluation and system development.

Summary updated

Abstract

Background The web has become a primary information resource about illnesses and treatments for both medical and non-medical users. Standard web search is by far the most common interface to this information. It is therefore of interest to find out how well web search engines work for diagnostic queries and what factors contribute to successes and failures. Among diseases, rare (or orphan) diseases represent an especially challenging and thus interesting class to diagnose as each is rare, diverse in symptoms and usually has scattered resources associated with it. Methods We design an evaluation approach for web search engines for rare disease diagnosis which includes 56 real life diagnostic cases, performance measures, information resources and guidelines for customising Google Search to this task. In addition, we introduce FindZebra, a specialized (vertical) rare disease search engine. FindZebra is powered by open source search technology and uses curated freely available online medical information. Results FindZebra outperforms Google Search in both default set-up and customised to the resources used by FindZebra. We extend FindZebra with specialized functionalities exploiting medical ontological information and UMLS medical concepts to demonstrate different ways of displaying the retrieved results to medical experts. Conclusions Our results indicate that a specialized search engine can improve the diagnostic quality without compromising the ease of use of the currently widely popular standard web search. The proposed evaluation approach can be valuable for future development and benchmarking. The FindZebra search engine is available at http://www.findzebra.com/ . Highlights ► Many clinicians use information retrieved from the web to aid the diagnostic process. Tools such as Google and PubMed are the preferred interfaces for seeking such information. ► We wanted to answer the question of how suitable these tools are in the diagnostic setting. In order to do that we sat up an evaluation approach. ► Made a dedicated easy-to-use rare disease search engine FindZebra to benchmark the web tools against. ► FindZebra is built upon open source information retrieval software and curated publicly available data. ► The evaluation performed on 56 difficult real life cases highlights the shortcomings of the web search engines and makes the case for the use of specialized search engines for specialized tasks.
BibTeX
@article{dragusin2013findzebra,
  author = {Radu Dragusin and Paula Petcu and Christina Lioma and Birger Larsen and Henrik Jørgensen and Ingemar J. Cox and Lars Kai Hansen and Peter Ingwersen and Ole Winther},
  title = {{FindZebra: A search engine for rare diseases}},
  journal = {Int. J. Medical Informatics},
  year = {2013},
  url = {https://openreview.net/forum?id=MA-IUlG97_},
}
Figure