Democratizing protein language models with parameter-efficient fine-tuning

Samuel Sledzieski, Meghana Kshirsagar, Minkyung Baek, Rahul Dodhia, Juan Lavista Ferres, Bonnie Berger

TL;DR

This paper introduces parameter-efficient fine-tuning (PEFT) methods, specifically LoRA, to protein language models (PLMs), making them more accessible to researchers with limited computational resources. PEFT achieves performance comparable to traditional full fine-tuning (FT) on protein-protein interaction (PPI) and homooligomer symmetry prediction tasks, while drastically reducing memory and parameter requirements.

What Problem Does This Paper Address?

Large protein language models (PLMs) have become powerful tools for understanding protein sequences, structures, and functions. However, adapting these massive models to specific downstream tasks typically requires "fine-tuning" (FT), which involves updating a significant portion of the model's parameters. This process is computationally intensive, demanding substantial GPU memory and processing power, thus creating a barrier for many academic labs, startups, and experimental groups with limited resources. This paper aims to bridge this gap by bringing parameter-efficient fine-tuning (PEFT) techniques, successfully employed in natural language processing (NLP), to the field of proteomics.

Methods

The core of this work is the application of Parameter-Efficient Fine-Tuning (PEFT) methods to PLMs. Specifically, the authors leverage LoRA (Low-Rank Adaptation), a popular PEFT technique. Instead of fine-tuning all the parameters of a PLM, LoRA injects small, trainable low-rank matrices into specific layers of the pre-trained model. Only these newly added matrices are updated during training, leaving the original PLM weights frozen. This drastically reduces the number of trainable parameters and, consequently, the computational and memory footprint.

The paper explores two main PEFT strategies:

  1. LoRA adaptation: Injecting low-rank adapter matrices into the transformer layers of the PLM.
  2. Classifier Head Only (MLP): Using the pre-trained PLM embeddings as-is and training only a simple multilayer perceptron (MLP) classifier on top.

They also conduct a thorough investigation into LoRA hyperparameters, particularly the choice of which weight matrices (query, key, value) to adapt and the rank of the adapter matrices.

Datasets

The paper utilizes existing benchmark datasets for its evaluations:

  • Protein-Protein Interaction (PPI) Prediction: The dataset from Bernett et al. (15) is used. This dataset is notable for its stringent splitting criteria (based on C3 similarity) and balanced positive/negative examples, aiming to prevent data leakage and biases related to sequence similarity or node degree.
  • Homooligomer Symmetry Prediction: Data from the Protein Data Bank (PDB) is used, with sequences clustered at 30% sequence identity to ensure robust evaluation. The dataset includes 17 symmetry classes plus an "Unknown" class.

Key Insights and Ideas

  • PEFT is a viable alternative to FT for PLMs: The study demonstrates that PEFT methods, particularly LoRA, can achieve performance comparable to traditional FT on complex proteomic tasks, making powerful PLMs more accessible.
  • Democratizing PLM adaptation: By significantly reducing computational requirements, PEFT empowers researchers with limited resources to leverage and adapt state-of-the-art PLMs.
  • LoRA's effectiveness in proteomics differs from NLP: While LoRA is generally robust, the optimal configuration for PLMs (e.g., which attention matrices to adapt and the recommended rank) may differ from NLP best practices. The paper provides a blueprint for these choices in proteomics.
  • MLP on frozen embeddings can be surprisingly effective: For PPI prediction, a simple MLP classifier trained on frozen PLM embeddings surprisingly outperformed even PEFT and FT models in some metrics, highlighting the power of pre-trained representations.
  • Deeper FT is enabled by PEFT: The reduced memory footprint of PEFT allows for fine-tuning of deeper layers of the PLM, which can lead to improved performance.

Key Results

  • Competitive Performance: PEFT models achieved performance competitive with, and in some cases exceeding, traditional FT models on both PPI and homooligomer symmetry prediction tasks.
  • Reduced Resource Usage: PEFT models required orders of magnitude fewer trainable parameters and significantly less GPU memory compared to FT. For instance, training only the classification head for PPI prediction used five orders of magnitude fewer parameters than full FT.
  • Outperforming State-of-the-Art: Both PEFT and MLP models outperformed existing state-of-the-art methods on the PPI prediction benchmark across several metrics.
  • LoRA Hyperparameter Insights: Adapting the key (WK) and value (WV) matrices of the transformer attention heads yielded the best performance for PPI prediction. A rank of at least 4 for LoRA adapters is recommended, though performance remains relatively robust.
  • Homooligomer Symmetry Nuances: While PEFT slightly trailed FT for homooligomer symmetry prediction, it significantly outperformed the MLP baseline. The MLP's performance advantage in PPI prediction was less pronounced for rarer symmetry classes, where PEFT and FT showed greater strength.

Limitations and Future Work

  • Hyperparameter Sensitivity: While PEFT is generally robust, the paper notes that the optimal LoRA rank and adapted matrices might vary across different tasks. They recommend testing hyperparameters on a validation set.
  • Task-Specific PEFT: The authors suggest that while LoRA is effective, other PEFT methods (like adapters) or even specialized approaches like quantization (e.g., QLoRA) might be beneficial for specific proteomic tasks, especially those involving very long sequences or protein complexes.
  • Broader Applicability: The study focused on ESM2. Future work could explore the effectiveness of PEFT on other PLMs and for different biological sequence types (e.g., DNA).
  • Model Scale vs. Data: The paper hints that while larger models are beneficial, data availability and diversity might be a more significant bottleneck for performance improvements in PLMs, suggesting that increased and more diverse protein sequence data could drive further advancements.
  • Inference Efficiency: The MLP approach, with its frozen PLM embeddings, offers significant inference-time efficiency, making it valuable for large-scale prediction tasks.

Summary updated

Abstract

Proteomics has been revolutionized by large protein language models (PLMs), which learn unsupervised representations from large corpora of sequences. These models are typically fine-tuned in a supervised setting to adapt the model to specific downstream tasks. However, the computational and memory footprint of fine-tuning (FT) large PLMs presents a barrier for many research groups with limited computational resources. Natural language processing has seen a similar explosion in the size of models, where these challenges have been addressed by methods for parameter-efficient fine-tuning (PEFT). In this work, we introduce this paradigm to proteomics through leveraging the parameter-efficient method LoRA and training new models for two important tasks: predicting protein-protein interactions (PPIs) and predicting the symmetry of homooligomer quaternary structures. We show that these approaches are competitive with traditional FT while requiring reduced memory and substantially fewer parameters. We additionally show that for the PPI prediction task, training only the classification head also remains competitive with full FT, using five orders of magnitude fewer parameters, and that each of these methods outperform state-of-the-art PPI prediction methods with substantially reduced compute. We further perform a comprehensive evaluation of the hyperparameter space, demonstrate that PEFT of PLMs is robust to variations in these hyperparameters, and elucidate where best practices for PEFT in proteomics differ from those in natural language processing. All our model adaptation and evaluation code is available open-source at https://github.com/microsoft/peft_proteomics. Thus, we provide a blueprint to democratize the power of PLM adaptation to groups with limited computational resources.
BibTeX
@inproceedings{sledzieski2024democratizing,
  author = {Samuel Sledzieski and Meghana Kshirsagar and Minkyung Baek and Rahul Dodhia and Juan Lavista Ferres and Bonnie Berger},
  title = {{Democratizing protein language models with parameter-efficient fine-tuning}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2024},
  doi = {10.1073/pnas.2405840121},
}
Figure