Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models

Francesca-Zhoufan Li, Ava P. Amini, Yisong Yue, Kevin Yang, Alex X. Lu

Abstract Large pretrained protein language models (PLMs) have improved protein property and structure prediction from sequences via transfer learning, in which weights and representations from PLMs are repurposed for downstream tasks. Although PLMs have shown great promise, currently there is little understanding of how the features learned by pretraining relate to and are useful for downstream tasks. We perform a systematic analysis of transfer learning using PLMs, conducting 370 experiments across a comprehensive suite of factors including different downstream tasks, architectures, model sizes, model depths, and pretraining time. We observe that while almost all downstream tasks do benefit from pretrained models compared to naive sequence representations, for the majority of tasks performance does not scale with pretraining, and instead relies on low-level features learned early in pretraining. Our results point to a mismatch between current PLM pretraining paradigms and most applications of these models, indicating a need for better pretraining methods.

Introduction

In recent years, Protein Language Models (PLMs) such as ESM have emerged as the "foundation models" of biology. By training on massive databases of protein sequences using Masked Language Modeling (MLM)—essentially a "fill-in-the-blanks" task for amino acids—these models learn representations that excel at downstream tasks like predicting protein structure or stability. However, the field has largely operated on the assumption that "bigger is better," a paradigm borrowed from Natural Language Processing. This study investigates whether these models are learning a deep "biological grammar" or simply providing a sophisticated weight initialization, specifically asking: how do features learned during pretraining relate to practical laboratory tasks?

Experimental Framework

To answer this, the researchers conducted a massive "stress test" of transfer learning, running 370 experiments across several dimensions: architecture (Transformers vs. CNNs), scaling (Small, Medium, Large), depth (layer-by-layer analysis), and training time (early vs. late checkpoints).

Overview of the pretraining and transfer learning evaluation framework, including hypothesized performance trends.

The study utilized linear probing—keeping the pretrained model frozen and training only a simple output layer—to isolate the information contained within the PLM's representations. The benchmarks included structure prediction (SS3), global properties (Thermostability, Subcellular Localization), and protein engineering tasks (GB1 and AAV datasets from the FLIP benchmark).

The Alignment Gap: Structure vs. Function

A central finding of this work is the "alignment gap" between structural and functional tasks. Secondary structure prediction is highly "aligned" with the MLM pretraining objective; performance improves consistently as models grow deeper and train longer. In contrast, most functional tasks, such as viral viability or binding affinity, are "unaligned." While they benefit from pretraining compared to one-hot baselines, they quickly hit a performance plateau.

Performance on functional tasks (Thermostability, GB1, AAV) showing limited scaling with model size and pretraining loss compared to structural tasks.

Key Insights

  • The "First Five Layers" Rule: For many functional tasks, the most useful features are captured within the first 3–5 layers. Moving deeper into a 33-layer model often provides zero marginal benefit or even degrades performance.
  • Architecture vs. Pretraining: For tasks like subcellular localization, the performance gain stems almost entirely from the model's architecture (inductive bias) rather than the pretraining on millions of sequences.
  • Diminishing Returns on Scaling: For protein engineering tasks (AAV and GB1), smaller or partially trained models often perform as well as state-of-the-art giants. This suggests that simply scaling up MLM-based models may not solve the hardest protein design problems.

Conclusion

The study reveals that while MLM captures co-evolutionary and structural patterns effectively, it may be fundamentally limited in capturing the dynamic functional properties required for advanced protein engineering. Despite the architectural differences, there was little empirical difference between CNNs (CARP) and Transformers (ESM) in feature reuse. The results suggest a critical need for new pretraining objectives that explicitly incorporate structural data or functional annotations to bridge the gap between sequence modeling and functional protein design.

Summary updated

Abstract

Abstract Large pretrained protein language models (PLMs) have improved protein property and structure prediction from sequences via transfer learning, in which weights and representations from PLMs are repurposed for downstream tasks. Although PLMs have shown great promise, currently there is little understanding of how the features learned by pretraining relate to and are useful for downstream tasks. We perform a systematic analysis of transfer learning using PLMs, conducting 370 experiments across a comprehensive suite of factors including different downstream tasks, architectures, model sizes, model depths, and pretraining time. We observe that while almost all down-stream tasks do benefit from pretrained models compared to naive sequence representations, for the majority of tasks performance does not scale with pretraining, and instead relies on low-level features learned early in pretraining. Our results point to a mismatch between current PLM pretraining paradigms and most applications of these models, indicating a need for better pretraining methods.
BibTeX
@inproceedings{li2024feature,
  author = {Francesca-Zhoufan Li and Ava P. Amini and Yisong Yue and Kevin Yang and Alex X. Lu},
  title = {{Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=wdTiuvd0fR},
}
Figure