TL;DR
This paper introduces the first comprehensive ontology (LSFO) and annotated corpus (LSF200) for lifestyle factors (LSFs) in biomedical literature. These resources enable the development of named entity recognition (NER) systems, leading to the identification of over 300 million LSF mentions in scientific texts, paving the way for better understanding of lifestyle's impact on health.
What Problem Does This Paper Address?
Lifestyle factors (LSFs) like diet, exercise, and sleep are increasingly recognized as crucial determinants of individual health, especially for chronic diseases. However, the biomedical literature lacks a systematic way to identify and categorize these factors. This is due to the inherent diversity of LSFs, the absence of a standardized classification, and a lack of dedicated resources like ontologies and annotated datasets for computational analysis. Without these, it's difficult to extract and analyze LSF information at scale, hindering research into their health impacts.
Methods
The authors tackled this problem by developing two key resources and two corresponding NER systems:
- Lifestyle Factor Ontology (LSFO): A novel, hierarchical ontology was created to classify LSFs. It defines nine main categories (e.g., Nutrition, Socioeconomic factors, Physical activities) and extends to more granular subcategories. This ontology serves as a structured knowledge base for LSFs.
- LSF200 Corpus: A manually annotated corpus of 200 biomedical abstracts was created, with LSF mentions labeled according to the LSFO. This corpus is crucial for training and evaluating machine learning-based NER systems.
- Dictionary-based NER System: This system leverages the LSFO to create a dictionary of LSF terms. It then uses a pre-existing, fast dictionary-based NER tool (JensenLab tagger) to identify LSFs in text. This approach prioritizes precision by matching known terms.
- Transformer-based NER System: An adaptation of a state-of-the-art transformer model (ROBERTa-bio) was trained on the LSF200 corpus. This system learns to identify LSFs based on their context in text, aiming for higher recall.
The dictionary construction involved several expansion stages, starting with manual curation and then incorporating sources like Wikipedia titles, WordNet, Wikidata, and other ontologies, with manual validation at each step.
Datasets
- LSF200 Corpus: A newly introduced, manually annotated corpus of 200 biomedical abstracts. It contains 1876 annotated LSF mentions across nine categories, plus an "Out of Context" (OOC) category to flag mentions that don't fit the LSF definition in a given context.
- PubMed Abstracts and PMC Open Access Articles: Large-scale datasets (36.1 million PubMed abstracts and 4.5 million PMC OA articles) were used to apply the developed NER systems and identify millions of LSF mentions.
Key Insights and Ideas
- The "Out of Context" (OOC) challenge: Identifying LSFs is complicated by words that can be LSFs in one context but not another (e.g., "tobacco" as a substance use vs. "tobacco company"). The LSF200 corpus includes OOC annotations to help address this, and the authors found that handling OOC mentions significantly impacts the precision of dictionary-based NER.
- Ontology quality matters: The authors found that only a small fraction of false negatives (missing LSF mentions) were due to missing concepts in the LSFO itself. This suggests the ontology has good breadth, and most missing mentions are due to fine-grained concepts or synonyms not yet captured.
- Precision vs. Recall Trade-off: The dictionary-based NER system achieves high precision but lower recall, while the transformer-based system offers better recall at the cost of slightly lower precision. This highlights the inherent trade-off between these two approaches.
- Transformer models excel at recall: The transformer-based NER system showed significant improvements in recall across most LSF categories, demonstrating its ability to generalize and identify LSFs even when they are not explicitly in a predefined dictionary.
Key Results
- LSFO and LSF200: The creation of a comprehensive LSF ontology and a manually annotated corpus are significant contributions to the field.
- NER Performance:
- Dictionary-based NER: Achieved an F-score of 64% on the LSF200 corpus.
- Transformer-based NER: Achieved an F-score of 76% on the LSF200 corpus.
- Large-scale application: Over 300 million LSF mentions were identified in PubMed and PMC OA articles using both systems.
- Error Analysis: The dictionary-based system's main errors were false negatives (missed mentions) due to abbreviations/synonyms and ambiguous names. The transformer-based system's errors were more often "model errors" or annotation errors.
Limitations and Future Work
- Dictionary-based NER Recall: The dictionary-based system's low recall for certain categories (e.g., "Environmental exposures") could be improved by integrating more fine-grained concepts from external resources via the established Xrefs.
- Transformer-based NER Performance: The transformer system performed worse than the dictionary-based one for "Substance use" and "Sleep" categories, suggesting areas for further model refinement.
- Ambiguous Names: The authors intentionally excluded some ambiguous names to maintain precision in the dictionary-based system, leading to potential false negatives. Future work could explore strategies to handle ambiguity more effectively.
- Future Directions: The authors suggest that the developed resources (LSFO, LSF200, and the NER systems) are publicly available and can serve as a foundation for further research, including relation extraction and a deeper understanding of lifestyle's role in health.
Summary updated