Efficient Parameter Tuning of Large Protein Language Models for De Novo Protein Design

Jiawei Luo, Xianliang Liu, Jiahao Li, Junjie ChenOpenReview 2024 · Submission to ICLR 2024 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Protein language models (ProtLMs) have achieved unprecedented breakthroughs in protein design. However, optimizing ProtLMs effectively with limited data has been challenging due to their large number of parameters. In this study, we introduce prefix tuning to efficiently prompt the pre-trained ProtLMs for de novo protein design with desired structures and functions. During the training process, only the prefix virtual token is trainable, while the pre-trained ProtLM is frozen. We trained two prefix virtual tokens on antimicrobial peptide (AMP) dataset and alpha-helix strucutre dataset, respectively. Our results demonstrate that prefix tuning is efficient to prompt the pre-trained ProtLM by optimizing fewer trainable parameters to achieve superior results compared with fine tuning, even under low-data settings. Furthermore, these two prefix virtual tokens can be combined to precisely control protein generation with both desired properties, which is not possessed by other tuning methods. We anticipate that prefix tuning will contribute to the protein discovery and biomedical advancement.
BibTeX
@misc{luo2024efficient,
  author = {Jiawei Luo and Xianliang Liu and Jiahao Li and Junjie Chen},
  title = {{Efficient Parameter Tuning of Large Protein Language Models for De Novo Protein Design}},
  year = {2024},
  url = {https://openreview.net/forum?id=hEGcVa1l4I},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}
Figure