Literature Mining for Incorporating Inductive Bias in Biomedical Prediction Tasks (Student Abstract)

Qizhen Zhang, Audrey Durand, Joelle PineauAAAI 2020 · DOI

Introduction

Machine learning (ML) models in the biomedical domain often suffer from "unrepresentative" datasets. A prominent example is the "healthy volunteer effect," where study participants are typically more health-conscious than the general population. When training data is biased—for instance, failing to capture the known correlation between high blood pressure and heart disease—standard models will learn these inaccuracies as ground truth. Traditional regularization techniques like L1 or L2 can prevent overfitting but cannot compensate for missing or distorted information within the dataset.

This research proposes a novel method to bridge this gap by leveraging the collective intelligence found in scientific literature. By mining millions of papers in PubMed, the authors create an inductive bias—a set of prior expectations—that guides models toward features known to be clinically significant, even when those relationships are obscured in the specific training data.

Methodology

The proposed framework incorporates expert knowledge into the model’s training phase through a two-step process:

1. Literature Mining

The researchers query PubMed using Medical Subject Headings (MeSH) relevant to the target prediction task. Using the quickUMLS tool, they extract medical keywords corresponding to each feature in the dataset. They then calculate a "relevancy score" ($α$) based on how frequently these keywords appear in the titles and abstracts of relevant papers. This score serves as a proxy for the scientific community's consensus on a feature's importance.

2. Regularization with Inductive Bias

The relevancy scores are normalized into a probability distribution ($Z$), while the model’s internal weights are transformed into a corresponding distribution ($˜{Z}$). The authors introduce a Kullback-Leibler (KL) divergence penalty to the loss function. This penalty mathematically "nudges" the model to align its internal feature priorities with the priorities established in the literature. If the model attempts to ignore a feature that the literature identifies as vital, it incurs a heavy penalty during training.

Experimental Design

To evaluate the method, the study utilized a cardiovascular disease prediction dataset (Ulianova 2019) containing 70,000 records and 11 features. The researchers designed a synthetic "worst-case scenario" experiment by intentionally manipulating the training data to hide the relationship between Systolic Blood Pressure (SBP)—a primary predictor of heart disease—and the outcome. This allowed them to test whether the model could "remember" the importance of SBP from the literature despite its absence in the data.

Key Results

The results of the synthetic experiment demonstrate the limitations of purely data-driven approaches. Standard models, including those utilizing L1 regularization, failed to identify the importance of SBP, resulting in poor predictive accuracy. In contrast, the model incorporating inductive bias from literature mining successfully identified SBP as a critical feature.

Comparison of model performance and feature weighting in the synthetic "hidden SBP" experiment.

As shown in the table above, the inductive bias model achieved an accuracy of 0.748, compared to 0.323 for standard models. Furthermore, it correctly assigned a positive weight (0.127) to SBP, whereas other models assigned near-zero or negative weights, effectively ignoring the feature.

Discussion and Future Work

By treating the body of medical literature as a quantifiable prior distribution, this approach moves beyond purely data-driven ML toward "knowledge-driven" models. This alignment not only improves performance on biased datasets but also enhances interpretability. Clinicians are more likely to trust a model that prioritizes features they know to be important.

However, the method faces certain limitations. It relies on descriptive feature identifiers; obscure coding in datasets can hinder the literature mining process. Additionally, the approach is dependent on the availability of relevant MeSH terms in PubMed. Future research will explore the use of the Wasserstein metric as an alternative to KL divergence, which may provide more robust mathematical properties for comparing literature distributions to model weights.

Summary updated

BibTeX
@inproceedings{zhang2020literature,
  author = {Qizhen Zhang and Audrey Durand and Joelle Pineau},
  title = {{Literature Mining for Incorporating Inductive Bias in Biomedical Prediction Tasks (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7264},
}

Loading related content...

Figure