ProtFIM: Fill-in-Middle Protein Sequence Design via Protein Language Models

Youhan Lee, Hasun YuICLR 2023 · OpenReview

TL;DR

This paper introduces ProtFIM, a novel protein language model designed for "fill-in-middle" protein sequence design. By considering both prefix and suffix sequence information, ProtFIM outperforms existing models in generating functional protein sequences with preserved secondary structures, a crucial aspect of real-world protein engineering.

What Problem Does This Paper Address?

Protein engineering aims to modify protein sequences to enhance their function. A common scenario in this field is optimizing amino acids within the middle of a protein sequence while keeping the rest of the sequence, and importantly, its secondary structure, intact. Existing protein language models (PLMs), primarily trained with a left-to-right (auto-regressive) approach, are good at generating sequences based on a preceding context but struggle with this "fill-in-middle" task. They lack the ability to effectively leverage information from both before and after the target region, potentially leading to suboptimal or even detrimental sequence designs. This paper addresses this gap by proposing a new approach and evaluation framework specifically for fill-in-middle protein sequence design.

Methods

The core innovation is the ProtFIM model, which is trained using a fill-in-middle (FIM) transformation. This technique, inspired by similar methods in natural language processing, involves randomly masking a contiguous span of residues in a protein sequence, moving it to the end, and then training the model to predict this masked span using both the preceding (prefix) and succeeding (suffix) context. This is achieved by introducing special tokens [PRE], [MID], and [SUF] to signal the different parts of the sequence during training. For inference, ProtFIM constructs a prompt by concatenating the prefix, suffix, and the [MID] token to guide the generation of the middle section.

To evaluate models on this specific task, the authors introduce SEIFER (Secondary structureE InFilling rEcoveRy). This benchmark assesses a model's ability to generate new protein sequences that not only differ from the original but also maintain the original secondary structure. This is crucial because secondary structures are fundamental to protein function and are often preserved in targeted engineering efforts. SEIFER uses AlphaFold2 to predict the 3D structures of generated sequences and then DSSP to determine their secondary structures, comparing them to the original.

Datasets

  • UniRef50: This dataset was used for training the ProtFIM model. The authors specifically mention using a version dated March 28, 2018, to avoid leakage from CASP13 and 14 datasets and ensure fair comparison with other models.
  • CASP14: Proteins from the CASP14 dataset were used to identify candidate middle sites for the SEIFER evaluation tasks.
  • PDB (Protein Data Bank): Used to extract 3D protein structures for analyzing the distribution of interaction sites and secondary structures.

Key Insights and Ideas

  • The "fill-in-middle" problem is critical for protein engineering: Many real-world protein engineering tasks require modifying internal residues while preserving overall structure and function, a task poorly suited to traditional auto-regressive language models.
  • Secondary structure preservation is key: Simply generating novel sequences isn't enough; maintaining the protein's secondary structure is vital for its function, making SEIFER a more relevant evaluation metric than simple sequence recovery.
  • Alpha-helix bias in training data: Natural protein datasets show a strong dominance of alpha-helices. This bias might explain why models perform better on alpha-helix generation but struggle with beta-sheets and coils, as they have less exposure to diverse patterns for these structures.
  • FIM training enhances context utilization: By explicitly training models to consider both prefix and suffix information, ProtFIM demonstrates a superior ability to leverage the full context for sequence generation.

Key Results

  • ProtFIM outperforms existing models on SEIFER: ProtFIM achieved better retrieval and precision scores on the SEIFER benchmark compared to auto-regressive PLMs like ProtGPT2-C and even larger models like ProGen2 and ProtXLNet, especially for alpha-helix structures.
  • Efficiency and effectiveness: ProtFIM, with 85M parameters, achieved comparable or better performance than models with significantly more parameters (e.g., ProGen2-medium at 764M).
  • Improved protein representations: ProtFIM showed strong performance in zero-shot fitness prediction tasks (FLIP benchmarks), indicating that its infilling approach leads to better protein representations that capture sequence-function relationships.
  • Enhanced structural integrity: ProtFIM generated sequences that often resulted in an increase in pLDDT scores (a measure of predicted structural confidence) compared to the original sequences, suggesting improved or well-maintained structures.
  • Random generators can be surprisingly competitive for short, simple regions: For very short or less structurally constrained middle regions, a random generator can perform comparably to sophisticated PLMs, highlighting the importance of the SEIFER metric for evaluating true engineering capability.

Limitations and Future Work

  • Limited dataset size for SEIFER: The SEIFER evaluation dataset, derived from 28 CASP proteins, is relatively small, which might explain some performance variations across different positions.
  • Alpha-helix bias: The paper acknowledges the alpha-helix bias in natural protein datasets and suggests that developing preprocessing or encoding techniques to alleviate this bias could be a future research direction.
  • Performance on coils and beta-sheets: While ProtFIM shows improvement, its performance on coil and beta-sheet structures is still comparable to or slightly better than random generators, suggesting room for improvement in capturing the complex rules governing these structures.
  • Future work: The authors propose ProtFIM as a tool for practical protein engineering, including fitness prediction and sequence optimization, and suggest its integration with structure predictors for a complete design framework. They also hint at future research into addressing the alpha-helix bias in training data.

Summary updated

Abstract

Protein language models (pLMs), pre-trained via causal language modeling on protein sequences, have been a promising tool for protein sequence design. In real-world protein engineering, there are many cases where the amino acids in the middle of a protein sequence are optimized while maintaining other residues. Unfortunately, because of the left-to-right nature of pLMs, existing pLMs modify suffix residues by prompting prefix residues, which are insufficient for the infilling task that considers the whole surrounding context. To find the more effective pLMs for protein engineering, we design a new benchmark, Secondary structureE InFilling rEcoveRy, SEIFER, which approximates infilling sequence design scenarios. With the evaluation of existing models on the benchmark, we reveal the weakness of existing language models and show that language models trained via fill-in-middle transformation, called ProtFIM, are more appropriate for protein engineering. Also, we prove that ProtFIM generates protein sequences with decent protein representations through exhaustive experiments and visualizations.
BibTeX
@inproceedings{lee2023protfim,
  author = {Youhan Lee and Hasun Yu},
  title = {{ProtFIM: Fill-in-Middle Protein Sequence Design via Protein Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=9XAZBUfnefS},
}
Figure