TL;DR
This paper investigates what factors most influence the performance of domain-adapted Large Language Models (LLMs) in the biomedical field. By employing a multilevel modeling approach, they reveal that both the intrinsic characteristics of the training data (like clarity and biomedical information density) and the choice of hyperparameters play equally crucial roles in achieving high downstream performance. This work offers a systematic way to understand and optimize LLM adaptation for specialized domains.
What Problem Does This Paper Address?
Adapting powerful general-purpose LLMs like Llama 2 to specialized domains, such as biomedicine, is crucial for unlocking their full potential in specific applications. However, this adaptation process, particularly Domain Adaptive Pretraining (DAPT), often leads to significant variability in downstream task performance. The authors observed this firsthand when adapting Llama 2 to biomedical text, where different subsets of PubMed abstracts and varying hyperparameter settings resulted in drastically different outcomes. The core problem this paper addresses is the lack of a clear understanding of why this variability occurs and what specific factors contribute most to successful domain adaptation. Without this understanding, optimizing the adaptation process remains largely a trial-and-error endeavor.
Methods
The authors employ a Multilevel Modeling (MLM) approach, a statistical technique commonly used in social sciences to analyze data with hierarchical structures. In this context, the "groups" are the different samples of PubMed abstracts used for DAPT, and the "individuals" within each group are the fine-tuned models generated from each sample with varying hyperparameters.
Their MLM model aims to predict downstream scores (specifically, Pearson correlation scores on the BIOSSES dataset) based on two levels of variables:
- Level-1 Variables: These relate to the choices made during the DAPT and fine-tuning processes, primarily hyperparameter combinations. They use indicator variables to represent different settings for random seeds, batch sizes, etc.
- Level-2 Variables: These are text features extracted from the samples of PubMed abstracts used for DAPT. These features are designed to capture intrinsic qualities of the text.
To handle potential multicollinearity (where independent variables are highly correlated), they use a multi-step variable selection strategy involving lasso regression, variance decomposition, and backward search. They also standardize independent variables for better comparison.
Datasets
- Training Data for DAPT: Subsets of PubMed abstracts. The paper specifically mentions using 70 different samples.
- Downstream Task Dataset: The BIOSSES dataset (Soğancıoğlu et al., 2017) for sentence similarity in the biomedical domain.
- Evaluation: The models are evaluated on their performance on the BIOSSES dataset, using Pearson correlation scores.
Key Insights and Ideas
- Data Quality and Hyperparameters are Equal Partners: The most striking insight is that the intrinsic features of the domain-specific text used for DAPT (like clarity and biomedical information density) and the choice of hyperparameters for both DAPT and fine-tuning contribute equally to downstream performance. This challenges the notion that one might be significantly more dominant than the other.
- Clarity and Simplicity Matter: Text features related to writing style, such as syntactic dependencies (specifically, the presence of adnominal clauses 'acl' which indicate clarity) and the avoidance of complex terms (like multiword expressions 'mwe' and 'cc.preconj'), significantly improve downstream scores. Conversely, overly complex or unclarity-inducing structures negatively impact performance.
- Biomedical Information Density is Key: A higher proportion of biomedical terms within the DAPT samples positively correlates with better downstream performance, suggesting that exposing the LLM to more domain-specific vocabulary is beneficial.
- The "Curse" of Multicollinearity: The authors highlight that many potential features are highly correlated, making it difficult to isolate their individual effects. Their rigorous selection process is crucial for disentangling these influences.
- MLM as a Powerful Tool for LLM Analysis: The paper advocates for the adoption of Multilevel Modeling in NLP research to gain a deeper understanding of complex model behaviors and the interplay of various influencing factors.
Key Results
- The Multilevel Model achieved an RMSE of 4.02 points, indicating that it can predict downstream scores with an average deviation of only 4 points from the actual scores.
- Biomedical_info had a positive standardized coefficient of 1.78, meaning a one standard deviation increase in biomedical term frequency leads to an average increase of 1.78 points in downstream scores.
- Parataxis (a feature indicating less clear sentence structure) had a negative effect, reducing scores by 1.92 points per standard deviation increase.
- Avg_word (average word length) had the biggest negative impact, with a coefficient of -3.91, suggesting that longer words significantly decrease scores.
- Hyperparameter combinations also showed significant impact, with HCF_5 leading to an average decrease of 2.76 points.
- Figure 3 indicates that approximately 55% of the variation in downstream scores is explained by the model's fixed and random effects, with text features and hyperparameters accounting for a substantial portion of this explained variance.
Limitations and Future Work
- Unexplained Variance: Despite the model's effectiveness, approximately 45% of the variation in downstream scores remains unexplained. This suggests that other factors, not captured by the current features or model structure, also play a role.
- Model Complexity: The authors acknowledge that a 3-level MLM might offer a more nuanced analysis but would require a significantly larger number of DAPT and fine-tuned models, posing computational challenges.
- Future Work: The paper suggests further analysis into how complex biomedical terms might paradoxically affect domain adaptation, despite their importance. They also emphasize the need for better document sampling strategies and hyperparameter search methods, informed by their findings.
Summary updated