LSD600: the first corpus of biomedical abstracts annotated with lifestyle–disease relations

Esmaeil Nouranı, Evangelia-Mantelena Makri, Xutao Mao, Sampo Pyysalo, Søren Brunak, Katerina Nastou, Lars Juhl JensenmedRxiv 2024 · DOI

TL;DR

Researchers have introduced LSD600, the first comprehensive dataset specifically designed to train AI to understand the complex relationships between lifestyle factors (like diet, exercise, or pollution) and diseases. By training a RoBERTa-based transformer model on this corpus, they’ve moved beyond simple "these two things are mentioned together" logic to accurately classifying whether a lifestyle factor causes, prevents, or treats a specific condition.

What Problem Does This Paper Address?

We know that lifestyle factors (LSFs) are critical to health, but most of that knowledge is buried in millions of unstructured scientific abstracts. Traditionally, researchers used "co-occurrence" models—if "coffee" and "cancer" appear in the same sentence, the model assumes they are related.

However, lifestyle relations are rarely that simple. For example, "exposure to radiation" can cause skin cancer, but "radiotherapy" (also radiation) is used to treat cancer. Simple models can't tell the difference. Furthermore, existing datasets were too narrow, focusing almost exclusively on nutrition rather than the full spectrum of lifestyle factors. There was a desperate need for a high-quality, "gold-standard" dataset to train models that understand context and nuance.

Methods

The authors developed a rigorous pipeline to create and validate the LSD600 corpus:

  1. Document Selection: They pulled 600 abstracts from PubMed, ensuring a diverse mix of nine lifestyle categories (nutrition, physical activity, environmental exposure, etc.).
  2. Manual Annotation: Using the BRAT Rapid Annotation Tool, experts manually labeled 1,900 relations. Crucially, they didn't just look at single sentences; 16% of the relations identified were cross-sentence, a much harder task for NLP models.
  3. The Model: They fine-tuned a RoBERTa-large model (a transformer-based architecture) for multi-label relation extraction.
  4. The "Masking" Trick: To ensure the model learned the context of a relationship rather than just memorizing specific entity names, they replaced disease and LSF mentions with "unused" generic tokens during training. This forces the model to look at the surrounding verbs and syntax to determine the relation type.

Datasets

  • LSD600 (New): The primary contribution, containing 600 abstracts, 5,027 disease mentions, and 6,930 LSF mentions.
  • LSF200: A previous dataset used as a starting point for some abstracts.
  • ND (Nutrition and Disease) & FoodDisease: Two external, nutrition-focused datasets used to validate how well the LSD600-trained model generalizes to other data.

Key Insights and Ideas

  • A Hierarchy of Relations: The authors defined eight distinct relation types (e.g., Causes, Prevents, Treats, No Association). This allows for much more granular knowledge graphs than previous "Related vs. Unrelated" binary models.
  • The Cross-Sentence Challenge: Most biomedical datasets limit themselves to relations within a single sentence. By including cross-sentence relations, LSD600 reflects the reality of scientific writing, where a factor might be introduced in one sentence and its effect on a disease described in the next.
  • Context is King: The model's ability to distinguish between Positive Statistical Association and Causes highlights the difficulty of the task. "Association" is common in observational studies, while "Causality" requires much stronger linguistic cues, which the model was able to pick up.

Key Results

  • Internal Performance: The RoBERTa model achieved an F-score of 68.5% on the held-out LSD600 test set.
  • Generalization: When tested on external datasets without any additional training, the model performed even better, hitting F-scores of 70.7% (ND corpus) and 80.7% (FoodDisease corpus). This suggests that the patterns learned from LSD600 are robust and applicable across the broader biomedical literature.
  • High Precision: On external datasets, the model showed exceptionally high precision (up to 94.3%), meaning that when the model claims a relationship exists, it is almost certainly correct.

Limitations and Future Work

The biggest hurdle remains cross-sentence relations. While the model is better than previous iterations, it still missed 61 out of 77 cross-sentence relations in the test set. The authors also noted that "convoluted text"—sentences with complex, nested clauses—remains a significant source of error for the AI.

Future work will likely involve integrating this model into larger pipelines to build automated knowledge graphs, helping researchers and clinicians quickly synthesize how various lifestyle interventions might impact specific patient populations.

Summary updated

Abstract

Abstract Lifestyle factors (LSFs) are increasingly recognized as instrumental in both the development and control of diseases. Despite their importance, there is a lack of methods to extract relations between LSFs and diseases from the literature, a step necessary to consolidate the currently available knowledge into a structured form. As simple co-occurrence-based relation extraction (RE) approaches are unable to distinguish between the different types of LSF-disease relations, context-aware transformer-based models are required to extract and classify these relations into specific relation types. No comprehensive LSF–disease RE system existed, primarily due to the lack of a suitable corpus for developing it. We present LSD600, the first corpus specifically designed for LSF-disease RE, comprising 600 abstracts with 1900 relations of eight distinct types between 5,027 diseases and 6,930 LSF entities. We evaluated LSD600’s quality by training a RoBERTa model on the corpus, achieving an F-score of 68.5% for the multi-label RE task on the held-out test set. We further validated LSD600 by using the trained model on the two Nutrition-Disease and FoodDisease datasets, where it achieved F-scores of 70.7% and 80.7%, respectively. Building on these performance results, LSD600 and the RE system trained on it can be valuable resources to fill the existing gap in this area and pave the way for downstream applications.
BibTeX
@misc{nouran2024lsd,
  author = {Esmaeil Nouranı and Evangelia-Mantelena Makri and Xutao Mao and Sampo Pyysalo and Søren Brunak and Katerina Nastou and Lars Juhl Jensen},
  title = {{LSD600: the first corpus of biomedical abstracts annotated with lifestyle–disease relations}},
  year = {2024},
  doi = {10.1101/2024.08.30.24312862},
  howpublished = {medRxiv},
}
Figure