A Multilevel Analysis of PubMed-only BERT-based Biomedical Models

Vicente Sanchez Carmona, Shanshan Jiang, Bin DongNAACL 2024

TL;DR

This paper introduces a novel multilevel analysis approach to dissect the contributions of different BERT-based biomedical models (BioBERT, BiomedBERT, BioLinkBERT) beyond simple average scores. The findings reveal that random seeds significantly impact performance, and while BiomedBERT shows the largest overall improvement over vanilla BERT, BioLinkBERT's "Link function" is only beneficial for specific QA tasks, suggesting other factors drive its success.

What Problem Does This Paper Address?

The field of biomedical Natural Language Processing (NLP) has seen significant advancements thanks to pre-trained language models like BERT. Specifically, models like BioBERT, MSR BiomedBERT (referred to as BiomedBERT), and BioLinkBERT, trained on PubMed abstracts, have been instrumental in tasks like information extraction and question answering from vast biomedical literature. However, current evaluation methods, which typically rely on a single average score across various benchmarks (like the BLURB benchmark), fail to provide a granular understanding of why these models perform well. It's difficult to disentangle the true impact of a model's novel contributions from confounding factors such as random seeds, learning rates, or dataset-specific variations. This lack of detailed insight hinders progress by making it hard to attribute performance gains to specific architectural or training innovations.

Methods

The authors propose using Multilevel Analysis (MLA), a statistical technique commonly employed in psychology and social sciences, to address this evaluation gap. MLA allows for the analysis of hierarchical data, where observations at one level are nested within another. In this context, the "lowest level" (Level 1) consists of individual fine-tuned models, while the "upper level" (Level 2) comprises groups of models based on the pre-trained model and the downstream dataset.

The MLA model incorporates variables at both levels:

  • Level 1 variables: Fine-tuning attributes like learning rate, batch size, and number of epochs.
  • Level 2 variables: Pre-trained model characteristics (e.g., DAPT for BioBERT, Pretrain_PubMed for BiomedBERT, Link function for BioLinkBERT) and the specific downstream dataset.

This approach allows the researchers to model the downstream test scores while accounting for the influence of these various factors simultaneously, without requiring computationally expensive ablation studies for each potential variable. They fit a 2-level multilevel model using the ImerTest R package and employ strategies for model building and hypothesis testing.

Datasets

The study utilizes the BLURB benchmark suite for evaluation. The authors fine-tune four pre-trained models: BioBERT, BiomedBERT, BioLinkBERT, and vanilla BERT (as a baseline) on all datasets within the BLURB suite. The specific datasets mentioned include BC2GM, BC5_chem, BC5_disease, NCBI, JNLPBA, PICO, ChemProt, DDI, GAD, BIOSSES, HOC, BioASQ, and PubMedQA.

Key Insights and Ideas

  • Random Seeds Matter (A Lot): Contrary to the assumption that random seeds are minor noise, this study reveals they can have a substantial impact on downstream scores, introducing significant variability. This highlights the importance of using multiple random seeds for robust evaluation.
  • BiomedBERT's Strong Performance: The analysis suggests that BiomedBERT, which was pre-trained from scratch on PubMed abstracts, offers the largest average improvement over vanilla BERT compared to the other models.
  • BioLinkBERT's Link Function is Task-Specific: While BioLinkBERT is a strong performer, its novel "Link function" contribution doesn't consistently boost performance across all tasks. Its effectiveness is primarily observed on Question Answering (QA) datasets like BioASQ and PubMedQA, indicating its utility is not universal.
  • Beyond Average Scores: The paper strongly advocates for more sophisticated evaluation methods like MLA, arguing that simple mean scores obscure crucial details about model contributions and interactions with different datasets.

Key Results

  • Fixed Effects:
    • BiomedBERT's pretraining ("Pretrain_PubMed") contributes an average gain of 4.36 points over vanilla BERT.
  • BioBERT's Domain Adaptive Pretraining (DAPT) contributes an average gain of 2.25 points over vanilla BERT.
  • BioLinkBERT's "Link function" shows a significant positive effect only on QA datasets (BioASQ and PubMedQA), leading to an average gain of 8.62 and 3.68 points, respectively, but has a negligible effect (0.07 points) on average across all datasets.
  • Random seeds (seed_20 and seed_47) introduce considerable variability, with scores varying by up to 4.79 and 6.21 points, respectively.
  • Random Effects:
    • Random coefficients for seeds and learning rates show that these variables behave differently across different pre-trained model and dataset combinations.
  • For example, BioBERT's performance can decrease by 2.33 and 3 points with specific random seeds, while BiomedBERT and BioLinkBERT might see slight gains.
  • R-squared: Fixed and random effects of the studied variables account for approximately 80% of the variance in test scores. Level-1 random coefficients (slope variation) contribute about 10% of this variance.

Limitations and Future Work

  • Unmeasured Variables: The authors acknowledge that other independent variables, such as detailed pretraining hyperparameters or data preprocessing methods, might influence downstream scores but were not included due to measurement difficulties or collinearity issues.
  • Hierarchy Design: The 2-level hierarchy might not be the most optimal design, and other hierarchical structures could be explored.
  • Model Sizes: The analysis focused on base-sized models (e.g., BioLinkBERT-base) and did not include larger variants (e.g., BioLinkBERT-large), which could exhibit different performance characteristics.
  • Batch Size: Due to GPU memory limitations, they could not explore larger batch sizes (e.g., batch size 64) for fine-tuning, which might benefit some models.
  • Future Work: The authors suggest exploring more independent variables and analyzing larger model variants. They also emphasize the need for the NLP community to adopt more sophisticated evaluation methods like MLMs for deeper insights.

Summary updated

BibTeX
@inproceedings{carmona2024multilevel,
  author = {Vicente Sanchez Carmona and Shanshan Jiang and Bin Dong},
  title = {{A Multilevel Analysis of PubMed-only BERT-based Biomedical Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure