Which pLM to choose?

Tobias Senoner, Ivan Koludarov, Joshua Guenther, Amarda Shehu, Burkhard Rost, Yana BrombergbioRxiv 2025 · DOI

Abstract

Protein-language models (pLMs) provide a novel means for mapping the protein space. Which of these new maps best advances specific biological analyses, however, is not obvious. To elucidate the principles of model selection, we benchmarked fourteen pLMs, spanning several orders of magnitude in parameter count, across a hundred million protein pairs, to assess how well they capture sequence, structure, and function similarity. For each model, we distinguish inherent information, i.e. signal recoverable from raw-embedding distances, and extractable information, i.e. signal revealed through additional supervised training. Three key results emerge. First, pLM protein representation space is inherently different from the space of biological protein representations, i.e. sequences or structures. Here, a size-performance paradox is salient – mid-scale foundation models are as good as much larger ones in reflecting all tested biological properties. Second, pLM representations compress and store biological information in proportion to model size. That is, a lightweight feed-forward network can be trained on embedding pairs to predict said biological properties well – a capacity dividend. Finally, we observe that a task-specific learning radically reshapes the embedding space, gaining inherent understanding of the task, but garbling any further extractions. In other words, smaller pLMs can provide efficient and compute-light general insight. Larger models are advantageous only when fine-tuning is planned to accomplish a specific task. Furthermore, representations generated by “specialist” models are not immediately generalizable throughout protein biology. Thus, for pLMs, bigger isn’t always better.

Introduction

Protein language models (pLMs) have revolutionized bioinformatics by treating amino acid sequences as a biological language. By mapping proteins into a high-dimensional "embedding space," these models allow researchers to analyze protein relationships without traditional alignment. However, the rapid proliferation of models—ranging from 8 million to over 15 billion parameters—has created a dilemma: which model is best for a given task? While Natural Language Processing (NLP) generally follows the rule that "bigger is better," this study investigates whether biological scaling laws follow the same trajectory.

Methods and Data

The researchers benchmarked 14 prominent pLMs, including the ESM family, ProtT5, Ankh, and ESM-C. The evaluation spanned 100 million protein pairs derived from SwissProt-pre2024. To test generalization, a New2024 set was used, containing proteins released in 2025 that were sequence-dissimilar to the training data.

The benchmark assessed how well embeddings capture three fundamental biological properties:

  1. Sequence Similarity: Measured via Percent Identity (PIDE).
  2. Structural Similarity: Measured via TM-score.
  3. Functional Similarity: Measured via the Homology-derived Functional Similarity of Proteins (HFSP) score.

The study distinguished between inherent information (accessible via simple Euclidean distances on raw embeddings) and extractable information (revealed by training a lightweight feed-forward network (FNN) on embedding pairs).

Results: The Size-Performance Paradox and Capacity Dividend

A central finding is the "size-performance paradox." When using raw embeddings (inherent information), larger models do not necessarily provide more accurate biological maps. In many cases, mid-scale models (300M to 650M parameters) perform as well as, or better than, their multi-billion parameter counterparts. For instance, the 8M parameter version of ESM-2 reflected functional similarity more accurately than the 3B version when using raw distances.

Figure 2: Pearson R2 correlation for sequence (A), structure (B), and function (C) tasks. Crosses indicate raw Euclidean distances (inherent signal), while circles indicate performance after supervised training (extractable signal).

Conversely, larger models exhibit a "capacity dividend." While their raw geometry might be "messy," they store significantly more biological information that can be unlocked through supervised learning. As shown in Figure 2, the performance of supervised "heads" (circles) scales much more consistently with parameter count than raw distances (crosses). For example, Ankh-Large (1.15B) showed a dramatic jump in extractable functional signal ($R^2 = 0.76$) compared to Ankh-Base (450M, $R^2 = 0.45$).

Discussion: Geometry and Specialization

The way these models distribute proteins in space varies significantly. Figure 5 illustrates the normalized pairwise distance distributions, showing how different architectures and training objectives "warp" the protein representation space. Specialist models, such as CLEAN (designed for enzyme function), radically reshape this space. While CLEAN excels at functional tasks, this specialization comes at a cost: it "garbles" other biological signals like structural similarity, making it less effective as a general-purpose tool.

Figure 5: Normalized pairwise distance distributions for various pLMs. The shifts in mean and variance highlight how different models perceive the "distance" between proteins in their embedding space.

Conclusion

The study concludes that for pLMs, "bigger isn't always better." Smaller and mid-sized models are often sufficient and more compute-efficient for general biological insights derived from raw embeddings. Massive models are primarily advantageous when researchers have the resources to perform task-specific supervised training to harvest the "capacity dividend." Performance ceilings (around $R^2 = 0.84$ for structure/function) suggest that future improvements may require more diverse training data or fundamental architectural shifts rather than simple scaling.

Summary updated

Abstract

A bstract Protein-language models (pLMs) provide a novel means for mapping the protein space. Which of these new maps best advances specific biological analyses, however, is not obvious. To elucidate the principles of model selection, we benchmarked fourteen pLMs, spanning several orders of magnitude in parameter count, across a hundred million protein pairs, to assess how well they capture sequence, structure, and function similarity. For each model, we distinguish inherent information , i.e. signal recoverable from raw-embedding distances, and extractable information , i.e. signal revealed through additional supervised training. Three key results emerge. First, pLM protein representation space is inherently different from the space of biological protein representations, i.e. sequences or structures. Here, a size-performance paradox is salient – mid-scale foundation models are as good as much larger ones in reflecting all tested biological properties. Second, pLM representations compress and store biological information in proportion to model size. That is, a lightweight feed-forward network can be trained on embedding pairs to predict said biological properties well – a capacity dividend. Finally, we observe that a task-specific learning radically reshapes the embedding space, gaining inherent understanding of the task, but garbling any further extractions. In other words, smaller pLMs can provide efficient and compute-light general insight. Larger models are advantageous only when fine-tuning is planned to accomplish a specific task. Furthermore, representations generated by “specialist” models are not immediately generalizable throughout protein biology. Thus, for pLMs, bigger isn’t always better.
BibTeX
@misc{senoner2025which,
  author = {Tobias Senoner and Ivan Koludarov and Joshua Guenther and Amarda Shehu and Burkhard Rost and Yana Bromberg},
  title = {{Which pLM to choose?}},
  year = {2025},
  doi = {10.1101/2025.10.30.685515},
  howpublished = {bioRxiv},
}
Figure