TL;DR
This paper explores whether reinforcement learning (RL) finetuning can enhance the biomedical reasoning capabilities of OpenAI's o1-mini large language model. While RL finetuning significantly boosts performance on gene classification tasks, outperforming even specialized models, its impact on small molecule property prediction is mixed, suggesting challenges in chemical reasoning for LLMs.
What Problem Does This Paper Address?
Large language models (LLMs) have shown remarkable progress in general reasoning tasks like math and coding. However, their effectiveness in specialized domains like biomedicine, which require deep domain knowledge and complex, open-ended reasoning, remains an open question. Existing LLMs, even those with advanced reasoning capabilities, don't consistently outperform simpler models on these biomedical tasks. This research aims to bridge this gap by investigating if RL finetuning can imbue LLMs with the necessary domain-specific reasoning to excel in biomedical applications.
Methods
The core of this research involves finetuning OpenAI's o1-mini model using reinforcement learning (RL). The authors compare the performance of the base o1-mini model ("o1-base") with its RL-finetuned counterpart ("o1-finetuned"). They also include comparisons with GPT-4o-mini (referred to as "4o-base" and "4o-finetuned" after supervised finetuning) as a baseline. To evaluate the models, they compare them against:
- Statistical Baselines: A majority class baseline (predicting the most frequent class) and an "all ones" baseline (predicting all instances as positive).
- Domain-Specific Models:
- For gene classification, they use GenePT, an embedding model trained on textual gene descriptions, with a logistic regression classifier.
- For small molecule property prediction, they use Chemprop-RDKit, a graph neural network (GNN) model specifically designed for molecular property prediction.
The finetuning process for o1-mini involved extracting the LLM-generated answer from a JSON object and comparing it to a reference answer using a string-check grader to compute a reward. For GPT-4o-mini, supervised finetuning was used, comparing the generated JSON string directly to a reference JSON string.
Datasets
The study utilizes two main categories of biomedical datasets:
Gene Functionality Classification: * Dosage Sensitivity: Distinguishing dosage-sensitive vs. insensitive transcription factors. * Promoter Classification: Categorizing promoter genes into bivalent, Lys4-only-methylated, or non-methylated. * Genome-wide Promoter Classification: Genome-wide profiling of promoter domains. * Long vs. Short-Range TFs: Distinguishing long-range vs. short-range transcription factors. * NOTCH1 Targets: Identifying NOTCH1-activated genes vs. non-targets. * NOTCH1 Centrality: Classifying genes as central vs. peripheral in the NOTCH1 network.
Small Molecule Property Prediction: * Therapeutics Data Commons (TDC) ADMET Benchmark: Six binary classification datasets covering properties like: * AMES: Mutagenicity. * Bioavailability: Oral bioavailability. * CYP2D6 Inhibition: Inhibition of CYP2D6 enzyme. * CYP3A4 Substrate: Metabolism by CYP3A4 enzyme. * CYP3A4 Inhibition: Inhibition of CYP3A4 enzyme. * P-gp: Inhibition of P-glycoprotein. * Antibiotics Dataset: A dataset of small molecules tested for inhibitory activity against Acinetobacter baumannii.
Key Insights and Ideas
- RL Finetuning Boosts Biomedical Reasoning: RL finetuning can significantly improve an LLM's ability to perform complex reasoning tasks in the biomedical domain, particularly for tasks involving structured biological knowledge.
- Gene Classification Success: RL finetuning of o1-mini led to substantial improvements in gene classification, with the finetuned model even outperforming specialized models like GenePT on some tasks. This suggests that LLMs, when properly finetuned, can integrate and leverage biological knowledge effectively.
- Chemical Reasoning is Challenging: The results for small molecule property prediction were mixed, indicating that chemical reasoning might be a more difficult challenge for LLMs compared to biological reasoning. This could be due to the different nature of data representations (SMILES vs. gene names) and the inherent complexity of chemical interactions.
- Class Imbalance is a Major Hurdle: A significant limitation observed is the tendency for finetuned models to "collapse" to predicting the majority class, especially on imbalanced datasets. This leads to high accuracy but lacks true predictive value, highlighting the need for robust methods to handle class imbalance in biomedical LLM finetuning.
- Interpretability Challenges Remain: Even when performance improves, the explanations generated by finetuned models can become less informative or even collapse to empty strings, making it difficult to understand how the model is reasoning. This poses a challenge for trusting and debugging these models.
Key Results
- Gene Classification:
- RL finetuned o1-mini achieved the highest average accuracy (0.75) and F1 score (0.75) across gene classification tasks, outperforming GenePT (0.74 accuracy, 0.72 F1) and majority class baselines.
- Finetuning for o1 yielded a larger improvement (average 8% accuracy increase) compared to 4o (less than 1% accuracy increase).
- Qualitative examples show that finetuned o1-mini provides more nuanced and accurate reasoning, correcting simplistic or incorrect inferences made by the base model.
- Small Molecule Property Prediction:
- Finetuning generally improved LLM accuracy and F1 scores on ADMET datasets.
- However, these improvements were often due to models predicting the majority class, rarely outperforming a simple majority class baseline or the domain-specific Chemprop GNN.
- In four out of six small molecule datasets, the finetuned o1-mini collapsed to predicting the majority class.
- For the antibiotics dataset, finetuned LLMs matched the majority class baseline in accuracy but significantly exceeded the "all ones" baseline in F1, especially after class balancing. However, they still did not outperform Chemprop when using SMILES as input.
Limitations and Future Work
- Class Imbalance Sensitivity: A major limitation is the finetuned models' tendency to collapse to majority class predictions on imbalanced datasets, which is problematic for true reasoning. Future work should focus on developing methods to mitigate this.
- Interpretability: The lack of clear reasoning trajectories and the reduction in explanation complexity post-finetuning make it difficult to ascertain if the model is truly reasoning more deeply or finding shortcuts.
- Domain Dependence: The efficacy of finetuning reasoning models appears to be highly dependent on the specific biomedical domain and dataset characteristics (data representation, size, class balance).
- Future Directions: The authors suggest exploring newer reasoning models (e.g., o3-mini) and further investigating the task and dataset conditions that lead to successful finetuning. This will help researchers decide when to use finetuned LLMs versus base LLMs or domain-specific tools.
Summary updated