Finding relevant biomedical datasets: the UC San Diego solution for the bioCADDIE Retrieval Challenge

Wei Wei, Zhanglong Ji, Yupeng He, Kai Zhang, Yuanchi Ha, Qi Li, Lucila Ohno‐MachadoDatabase 2018 · DOI

TL;DR

This paper presents a novel pipeline for retrieving relevant biomedical datasets across diverse repositories, addressing the challenge of fragmented data and complex search needs. Their system achieved the highest score in the bioCADDIE Retrieval Challenge by effectively augmenting dataset metadata, intelligently generating queries from user questions, and employing a sophisticated re-ranking strategy.

What Problem Does This Paper Address?

The explosion of biomedical data has led to numerous specialized repositories, each with its own format and metadata. This fragmentation makes it incredibly difficult for researchers to discover all relevant datasets, especially across different domains or repositories they might not be aware of. Existing search systems often struggle with cross-repository searches and understanding natural language queries. This paper tackles the challenge of building a unified system that can effectively search across these disparate sources, understand user intent expressed in free text, and deliver high-quality, relevant dataset results.

Methods

The UC San Diego team developed a five-module retrieval pipeline:

  1. Additional Data Collection: They enriched existing dataset metadata by gathering supplementary information from online resources like GEO, which provided "Summary," "Title," and "Overall design" fields. This aimed to compensate for limited original metadata.
  2. Dataset Indexing: The combined metadata and additional information were indexed using Elasticsearch for efficient searching.
  3. Query Generation: A key innovation was their ability to automatically generate Boolean queries from users' free-text questions. This involved removing non-informative words, using PubMed to expand key concepts (e.g., "TGF-beta" to "TGFbeta" and "transforming growth factor beta"), and then constructing nested Boolean queries.
  4. Dataset Retrieval: Elasticsearch was used for the initial retrieval of datasets based on the generated queries.
  5. Re-ranking: To improve the precision of the initial results, they employed a two-step re-ranking strategy. This involved a Pseudo-Sequential Dependence (PSD) model, a Snippet-based Query Expansion Method (SQEM), and an Ensemble method that combined the results of multiple re-ranking algorithms. The PSD model was particularly adapted to prioritize the existence of a word in metadata over its frequency.

Datasets

The primary dataset used for evaluation was provided by the bioCADDIE Retrieval Challenge. This collection consisted of metadata for 794,992 biomedical datasets from 20 repositories. The metadata followed the Data Tag Suite (DATS) model and was provided in XML and JSON formats. The challenge also included 30 example questions and 15 test questions derived from three use cases, with some questions having pre-determined relevance judgments. The authors also collected additional data for 158,963 datasets from Arrayexpress, Gemma, and GEO databases.

Key Insights and Ideas

  • Metadata Augmentation is Crucial: Simply relying on existing metadata is insufficient. Supplementing it with information from related sources significantly improves retrieval performance.
  • Natural Language Understanding is Key: The ability to translate free-text user questions into effective search queries is paramount for user accessibility and finding relevant data. Their query expansion using PubMed and concept mapping was a critical component.
  • Re-ranking is Essential for Precision: While initial retrieval aims for recall, re-ranking is vital for achieving high precision and delivering the most relevant results to the user.
  • Ensemble Methods Offer Robustness: Combining multiple re-ranking strategies can lead to more stable and improved performance compared to any single method.
  • Query Expansion is More Impactful than Additional Fields (in some contexts): Their breakdown analysis suggested that query expansion had a more significant impact on infNDCG than the inclusion of additional fields or standard fields alone, highlighting the importance of understanding user intent.

Key Results

The UC San Diego team's pipeline achieved the highest inferred Normalized Discounted Cumulative Gain (infNDCG) score in the bioCADDIE Retrieval Challenge among 45 submissions from 10 teams. Their PSD-allwords method specifically achieved the highest infNDCG and Precision@10(+partial). The Ensemble method and SQEM also performed very well, placing second and third for infNDCG, respectively. The breakdown analysis showed that including all three features (additional fields, standard fields, and query expansion) yielded the best infNDCG, with query expansion being particularly critical.

Limitations and Future Work

The authors acknowledge several limitations:

  • Lack of Metadata Normalization: Concepts in metadata and additional information were not normalized, potentially leading to missed matches (e.g., "TGF-beta" vs. "TGF beta").
  • Limited Feature Set in Re-ranking: Re-ranking algorithms did not consider complex features like named entities, which could improve disambiguation.
  • Potential for Ambiguity: Query expansion could be further improved by applying disambiguation methods to reduce the retrieval of irrelevant datasets.

Future work could explore:

  • Machine Learning for Query Generation: Using deep learning methods to translate questions into sentence embeddings could preserve more information and lead to more effective retrieval.
  • Enhanced Feature Engineering for Re-ranking: Incorporating more sophisticated features beyond unigrams could improve re-ranking performance.
  • More Labeled Data: The current test set size limits definitive conclusions about the superiority of certain methods, suggesting a need for more labeled questions to better understand trade-offs.

Summary updated

Abstract

https://github.com/w2wei/dataset_retrieval_pipeline.
BibTeX
@inproceedings{wei2018finding,
  author = {Wei Wei and Zhanglong Ji and Yupeng He and Kai Zhang and Yuanchi Ha and Qi Li and Lucila Ohno‐Machado},
  title = {{Finding relevant biomedical datasets: the UC San Diego solution for the bioCADDIE Retrieval Challenge}},
  booktitle = {Database},
  year = {2018},
  doi = {10.1093/database/bay017},
}
Figure