Information Retrieval Using Machine Learning for Biomarker Curation in the Exposome-Explorer

Andre Lamurias, Sofia Jesus, Vanessa Neveu, Reza M. Salek, Francisco M. Couto

TL;DR

This paper introduces a machine learning pipeline to automate the laborious process of curating the Exposome-Explorer database, which catalogs biomarkers of exposure. By training classifiers on existing curated data, they significantly reduced the manual screening effort by nearly 90% while identifying new candidate biomarkers.

What Problem Does This Paper Address?

Manually curating large scientific databases is a time-consuming and resource-intensive task, especially when dealing with vast amounts of literature. The Exposome-Explorer, a database of biomarkers of exposure, faced this challenge. It was initially built from over 8,500 citations identified through manual literature searches, but only a fraction of these were ultimately included. This manual process requires domain expertise and is difficult to scale as new research emerges. The paper aims to address this by developing an automated system to assist in identifying relevant publications and potential new biomarker entities, thereby accelerating the curation process and keeping the database up-to-date.

Methods

The core of this work is a supervised machine learning pipeline. They utilized existing curated data from the Exposome-Explorer as training and testing sets.

  • Document Classification: They trained various machine learning classifiers (Logistic Regression, Naïve Bayes, Decision Trees, Neural Networks, Random Forest, Support Vector Machines) to predict whether an article was relevant to the database. This involved experimenting with different text features derived from article titles, abstracts, and metadata. They also explored ensemble methods like Bagging and Stacking to combine predictions from multiple models.
  • Biomarker Entity Recognition (NER): After identifying relevant documents, they trained a NER model to automatically extract potential biomarker entities from the abstracts. This model was trained on automatically annotated data derived from the existing database, using a Transformer architecture fine-tuned for the biomedical domain.

Datasets

  • Exposome-Explorer Dataset: The primary dataset was derived from the Exposome-Explorer database itself. This included:
  • Search queries used in Web of Science (WOS) for dietary and pollutant biomarkers.
  • Approximately 8,500 citations initially retrieved.
  • 480 publications used to construct the original database.
  • PubMed Corpus: To augment the existing data, they created a corpus of 7,083 publications by extracting titles, abstracts, and metadata from PubMed using E-utilities. This corpus was then labeled based on its presence in the original Exposome-Explorer curated set (relevant = 1, irrelevant = 0).
  • Silver Standard Corpus for NER: For the biomarker entity recognition task, they created a "silver standard" corpus by automatically annotating biomarker names from the Exposome-Explorer database using a tool called MER. This dataset was used to train and evaluate the NER model.

Key Insights and Ideas

  • Abstracts and Titles are Key: The study found that combining article titles and abstracts provided the best performance for document classification, highlighting the importance of these sections for quickly assessing relevance. Metadata alone was less effective.
  • Dietary Biomarkers are Easier to Classify: The models performed significantly better when trained and tested on data specifically related to dietary biomarkers compared to a broader set including pollutants. This is attributed to more focused curation efforts and clearer correlation criteria for dietary biomarkers in the original database.
  • Automated NER for Candidate Discovery: The NER component proved effective in identifying potential new biomarker entities, with 45 novel candidates identified from documents not previously used in the database. This demonstrates the utility of automated entity extraction for expanding curated resources.
  • Trade-off Between Precision and Recall: The paper emphasizes the importance of balancing precision (minimizing false positives) and recall (minimizing false negatives). While high recall is desirable to avoid missing relevant papers, a high number of false positives can overwhelm curators. The F2-score was used to prioritize recall.

Key Results

  • Document Classification Performance: The top-performing classifier was a Logistic Regression model using titles and abstracts, achieving an F2-score of 70.1%. This model could reduce the number of articles to be manually screened by nearly 90%, while only misclassifying 22.1% of relevant articles.
  • Biomarker Entity Recognition: The NER classifier achieved an F1-score of 0.6735 on a held-out test set, with a precision of 0.5574 and recall of 0.8507.
  • New Candidate Biomarkers: The NER model identified 45 new candidate biomarker entities that were not previously in the Exposome-Explorer database.

Limitations and Future Work

  • Silver Standard for NER: The NER model was trained on automatically annotated data, which may introduce noise and affect precision. Future work could involve creating a true gold-standard dataset for more robust NER.
  • Metadata Underutilization: The performance of classifiers using metadata was generally lower. Future work aims to improve this by assigning weights to authors or creating new features from author combinations.
  • Generalizability to Other Biomarker Types: While the model performed well on dietary biomarkers, its performance on pollutant biomarkers and other types needs further investigation. Expanding the training data to include a more comprehensive corpus for various biomarker classes is suggested.
  • Deep Learning Exploration: The authors suggest exploring deep learning models for document classification, acknowledging that this would require more training data and computational resources.
  • User Study for Curator Preference: The paper notes that understanding the ideal balance between precision and recall for database curators would benefit from a user study.
  • Handling of Specific Terms: The analysis revealed that certain terms (e.g., "selenium") could lead to misclassification if over-represented in irrelevant documents. Future work may involve categorizing tokens (e.g., "chemical") to mitigate this.

Summary updated

Abstract

p><bold>Objective:</bold> In 2016, the International Agency for Research on Cancer, part of the World Health Organization, released the Exposome-Explorer, the first database dedicated to biomarkers of exposure for environmental risk factors for diseases. The database contents resulted from a manual literature search that yielded over 8,500 citations, but only a small fraction of these publications were used in the final database. Manually curating a database is time-consuming and requires domain expertise to gather relevant data scattered throughout millions of articles. This work proposes a supervised machine learning pipeline to assist the manual literature retrieval process.</p><p><bold>Methods:</bold> The manually retrieved corpus of scientific publications used in the Exposome-Explorer was used as training and testing sets for the machine learning models (classifiers). Several parameters and algorithms were evaluated to predict an article’s relevance based on different datasets made of titles, abstracts and metadata.</p><p><bold>Results:</bold> The top performance classifier was built with the Logistic Regression algorithm using the title and abstract set, achieving an F2-score of 70.1%. Furthermore, we extracted 1,143 entities from these articles with a classifier trained for biomarker entity recognition. Of these, we manually validated 45 new candidate entries to the database.</p><p><bold>Conclusion:</bold> Our methodology reduced the number of articles to be manually screened by the database curators by nearly 90%, while only misclassifying 22.1% of the relevant articles. We expect that this methodology can also be applied to similar biomarkers datasets or be adapted to assist the manual curation process of similar chemical or disease databases.</p>
BibTeX
@article{lamurias2021information,
  author = {Andre Lamurias and Sofia Jesus and Vanessa Neveu and Reza M. Salek and Francisco M. Couto},
  title = {{Information Retrieval Using Machine Learning for Biomarker Curation in the Exposome-Explorer}},
  journal = {Frontiers Res. Metrics Anal.},
  year = {2021},
  doi = {10.3389/frma.2021.689264},
}
Figure