TL;DR
This paper investigates whether using journal impact metrics to filter pre-training data for biomedical language models improves performance. They found that filtering based on journal impact metrics did not significantly outperform random selection, suggesting that simpler methods or different quality indicators might be more effective for domain adaptation.
What Problem Does This Paper Address?
Natural Language Processing (NLP) models, especially large ones like BERT, have shown remarkable success across various domains. However, to perform well in specialized fields like biomedicine, which has its own unique vocabulary and knowledge, these models often need to be adapted. This adaptation typically involves pre-training the model on a large corpus of domain-specific text. While PubMed is a common source for biomedical text, the sheer volume of data raises questions about how to best select subsets for pre-training. Specifically, this paper asks: can we improve biomedical NLP model performance by intelligently selecting scientific documents for pre-training using journal impact metrics, and does using fewer, high-quality abstracts still yield good results?
Methods
The researchers employed a methodology similar to prior work by Marion et al. (2023), focusing on creating subsets of a large dataset based on a specific metric. They used journal impact metrics – the h-index and the Scimago Journal Rank (SJR) – to score scientific papers. They then created subsets by selecting the top 25% and 50% of documents based on these metrics, as well as a random selection for comparison. The core idea was to continue pre-training a BERT-base model on these curated subsets. A key aspect of their approach was filtering documents before tokenization, as their metrics were applied at the document level. They then evaluated the performance of these adapted models on various biomedical language understanding tasks from the BLURB benchmark.
Datasets
- Training Data: The primary dataset used was the PubMed January 2024 baseline corpus, consisting of all article abstracts deposited on PubMed up to that date. This corpus was filtered to include only English abstracts with available text and valid journal identifiers.
- Evaluation Datasets: The models were evaluated on several tasks from the BLURB benchmark, specifically:
- Named Entity Recognition (NER): BC5-chem, BC5-disease, BC2GM, JNLPBA, and NCBI-disease.
- Relation Extraction: ChemProt, DDI, and GAD.
- Document Classification: HoC.
- Question Answering: PubMedQA and BioASQ Task 7b.
- Note: The PICO and Sentence Similarity tasks were excluded due to difficulties in reproducing results.
Key Insights and Ideas
- Journal Impact Metrics Aren't a Magic Bullet: The most striking finding is that filtering pre-training data using journal impact metrics (h-index and SJR) did not lead to significant performance improvements compared to random selection. This suggests that these metrics, while indicative of a journal's prestige, may not be the most effective indicators of abstract quality for model pre-training.
- Less Can Be As Good As More (Sometimes): The study observed that reducing the number of abstracts in the training set (e.g., using 25% vs. 50%) did not necessarily decrease model performance. This hints at the possibility that focusing on a smaller, potentially higher-quality subset might be as effective as using a larger, more diverse one, especially if the larger set contains more noise.
- The Need for Better Quality Indicators: The results highlight a gap in our understanding of how to best select pre-training data for specialized domains. The authors suggest that more appropriate metrics are needed to assess document quality, possibly at a different level (e.g., corpus-level) or by using full-text articles where document impact might be more pronounced.
Key Results
The paper reports that the best overall performance (in micro and macro averages) was achieved by the model trained on the top 50% of abstracts based on the h-index. However, the performance differences between models trained on journal-impact-filtered data, randomly selected data, and even smaller subsets were generally not significant. For instance, the micro average performance for the base BERT model was 77.07, while the best performing filtered model achieved 79.43. While this is an improvement, the paper emphasizes that the gains were marginal and not consistently superior across all tasks.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Limited Scope of Metrics: The study focused on only two journal impact metrics (h-index and SJR) and did not explore other potential quality indicators.
- Abstracts vs. Full Text: The experiments were conducted using abstracts. Future work could investigate the impact of using full-text articles, where document quality might be more discernible.
- Subset Size Exploration: The paper suggests further experiments with different subset sizes to determine the optimal number of documents for pre-training without sacrificing performance.
- Developing Better Metrics: A key future direction is to find more appropriate metrics or combinations of metrics to assess document quality for pre-training, potentially at the corpus level or by incorporating other domain-specific signals.
- Pre-training on the Full Set: Due to resource constraints, they did not perform a pre-training experiment on the non-filtered PubMed set. They plan to do this in future work.
Summary updated