One protein is all you need

Anton Bushuiev, Roman Bushuiev, Olga Pimenova, Raman Samusevich, H. Stärk, Jiří Sedlář, Rachel Hyunkyung Kim, Martin Steinegger, Tomáš Pluskal, Josef Sivic

TL;DR

This paper introduces Protein Test-Time Training (ProteinTTT), a novel method that allows protein language models to be customized to individual proteins on-the-fly. This customization significantly improves performance on tasks like protein structure prediction and fitness prediction, especially for challenging or novel proteins not well-represented in training data.

What Problem Does This Paper Address?

Machine learning models for protein analysis, like those for structure prediction or function prediction, are typically trained on massive datasets to achieve good average performance. However, biologists often need highly accurate predictions for specific, individual proteins that might be outliers or entirely novel, and for which training data is scarce. Existing models struggle to generalize to these specific cases, limiting their practical utility for experimentalists who need precise insights for their research. This paper addresses the challenge of adapting general-purpose protein models to excel on individual proteins without requiring additional training data.

Methods

The core innovation is Protein Test-Time Training (ProteinTTT). This method leverages the self-supervised pre-training objective of protein language models (PLMs), typically masked language modeling (MLM), to fine-tune the model specifically for a single target protein during inference.

Here's how it works:

  • Y-shaped Architecture: The paper assumes a common "Y-shaped" model architecture in protein ML, consisting of a pre-trained backbone (e.g., ESM2) and a downstream task-specific head (e.g., for structure prediction).
  • Self-Supervised Customization: ProteinTTT freezes the downstream head and uses the self-supervised head (e.g., MLM) to adapt the backbone parameters to the target protein's sequence. This is done for a small number of "customization steps."
  • Perplexity Minimization: The underlying hypothesis is that minimizing the model's perplexity (or "surprise") on a given protein sequence leads to better representations. By optimizing the MLM loss on the target protein, ProteinTTT effectively reduces its perplexity.
  • Confidence-Based Selection: For tasks like structure prediction where a confidence score (e.g., pLDDT) is available, ProteinTTT selects the best customization step based on this score, preventing overfitting.
  • Efficiency: To handle large models, ProteinTTT employs techniques like Low-Rank Adaptation (LoRA) and gradient accumulation, enabling customization on a single GPU with minimal overhead.

The paper also explores extensions for autoregressive models and models using multiple sequence alignments (MSAs).

Datasets

The paper utilizes several established datasets for evaluation:

  • CAMEO: A benchmark for protein folding, used for evaluating structure prediction. The authors specifically selected challenging targets with low ESMFold pLDDT scores or high ESM2 perplexity.
  • ProteinGym: A standard benchmark for protein fitness prediction, containing deep mutational scanning experiments.
  • MaveDB: A newly constructed dataset for protein fitness prediction, mined from Multiplexed Assays of Variant Effect (MAVEs). This dataset was used as a validation set to ensure it was disjoint from ProteinGym.
  • Big Fantastic Virus Database (BFVD): Used for expanding known structures of viral proteins.

Key Insights and Ideas

  • Perplexity as a Proxy for Accuracy: The paper provides a strong theoretical and empirical link between minimizing a protein's perplexity and improving downstream prediction accuracy. This is a key insight into why test-time customization works.
  • "One Protein is All You Need": The core idea is that focusing customization on a single protein at a time, rather than broad generalization, unlocks significant performance gains for specific, challenging cases.
  • Leveraging Pre-trained Models: ProteinTTT doesn't require retraining from scratch; it effectively "wakes up" the latent knowledge within existing, powerful PLMs for a specific target.
  • Efficiency is Key: The method is designed to be computationally efficient, making it practical for use on individual proteins without prohibitive costs.
  • Beyond Structure Prediction: While structure prediction is a primary focus, the paper demonstrates ProteinTTT's effectiveness across fitness and function prediction tasks, highlighting its broad applicability.

Key Results

  • Structure Prediction: ProteinTTT consistently improved structure prediction quality (measured by TM-score and LDDT) for models like ESMFold, HelixFold-Single, and ESM3, particularly on challenging targets. For example, it improved predictions for 19% of structures in the Big Fantastic Virus Database.
  • Fitness Prediction: ProteinTTT achieved new state-of-the-art results on the ProteinGym benchmark, setting a new record for Spearman correlation. It showed particular benefit for proteins with low MSA depth.
  • Function Prediction: ProteinTTT enhanced performance on terpene synthase substrate classification and protein localization prediction tasks.
  • Antibody-Antigen Loops: In a case study, ProteinTTT significantly improved the modeling of antibody-antigen loops (CDRs), achieving higher average LDDT scores.
  • Speed: ProteinTTT, when applied to ESMFold, maintained ESMFold's speed advantage and was an order of magnitude faster than AlphaFold2.

Limitations and Future Work

  • Failure Modes: The authors acknowledge that ProteinTTT's failure modes are not fully understood. In some cases, customization can lead to performance degradation, especially when the evaluation itself is ambiguous or the protein has inherent conformational variability.
  • Confidence Metrics: The effectiveness of ProteinTTT relies on having a reliable confidence metric (like pLDDT for structure prediction) to select the optimal customization step and mitigate overfitting. Such metrics are not universally available for all tasks (e.g., fitness, function prediction), making failure control more challenging.
  • Model Types: The current method is primarily designed for Transformer-based PLMs trained with masked language modeling or autoregressive objectives.
  • Future Work: The authors suggest extending ProteinTTT to other model types (e.g., MSA-based models like AlphaFold2), exploring its application in de novo protein design, and developing more robust, task-agnostic confidence measures or dedicated confidence prediction modules.

Summary updated

Abstract

Generalization beyond training data remains a central challenge in machine learning for biology. A common way to enhance generalization is self-supervised pre-training on large datasets. However, aiming to perform well on all possible proteins can limit a model's capacity to excel on any specific one, whereas experimentalists typically need accurate predictions for individual proteins they study, often not covered in training data. To address this limitation, we propose a method that enables self-supervised customization of protein language models to one target protein at a time, on the fly, and without assuming any additional data. We show that our Protein Test-Time Training (ProteinTTT) method consistently enhances generalization across different models, their sizes, and datasets. ProteinTTT improves structure prediction for challenging targets, achieves new state-of-the-art results on protein fitness prediction, and enhances function prediction on two tasks. Through two challenging case studies, we also show that customization via ProteinTTT achieves more accurate antibody-antigen loop modeling and enhances 19% of structures in the Big Fantastic Virus Database, delivering improved predictions where general-purpose AlphaFold2 and ESMFold struggle.
BibTeX
@inproceedings{bushuiev2026one,
  author = {Anton Bushuiev and Roman Bushuiev and Olga Pimenova and Raman Samusevich and H. Stärk and Jiří Sedlář and Rachel Hyunkyung Kim and Hannes Stärk and Jiri Sedlar and Martin Steinegger and Tomáš Pluskal and Josef Sivic},
  title = {{One protein is all you need}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=5zAde2jch7},
}
Figure