Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization

Amy X. Lu, Haoran Zhang, Marzyeh Ghassemi, Alan M. Moses

TL;DR

This paper introduces CPCProt, a self-supervised learning method for protein sequences that uses mutual information maximization to learn powerful embeddings. Remarkably, CPCProt achieves comparable performance to state-of-the-art models on various biological tasks while using significantly fewer parameters, making it more accessible and efficient.

What Problem Does This Paper Address?

The explosion of protein sequence data from advanced sequencing technologies has outpaced our ability to label it for supervised learning tasks. This creates a bottleneck for understanding protein function and structure. While self-supervised learning offers a promising avenue to learn informative representations without labels, existing methods often rely on complex architectures and large parameter counts, making them computationally expensive and less accessible. This paper aims to develop a more efficient self-supervised method for learning protein embeddings that captures essential biological information.

Methods

The core of CPCProt lies in applying the Contrastive Predictive Coding (CPC) framework to protein sequences. The principle is mutual information maximization (InfoMax), which aims to learn representations that capture as much information as possible about the input.

Here's how it works:

  • Patching: Protein sequences are divided into fixed-size "patches" (e.g., 11 amino acids).
  • Encoding: Each patch is encoded into a vector using a convolutional encoder (genc). These encoded vectors are concatenated to form a latent representation z.
  • Autoregression: An autoregressive model (gar) takes the latent representation z and predicts a "context" vector c that summarizes the global context of the protein sequence.
  • Contrastive Loss (InfoNCE): The model is trained to distinguish between a "real" future patch's embedding and a set of "fake" embeddings from other proteins in the batch. This is achieved by minimizing the InfoNCE loss, which provides a lower bound on the mutual information between the context c and future patch embeddings. This forces the model to learn representations that capture both local (patch) and global (context) information.

A key innovation is the use of a parameterless dot product critic and drawing negative samples in-batch, which simplifies the architecture and improves efficiency compared to some prior methods.

Datasets

  • Pretraining: The models were pretrained on a large dataset of Pfam domain sequences (over 32 million sequences).
  • Downstream Evaluation: The models were evaluated on several established protein-related downstream tasks, including:
  • Remote Homology: Classifying protein structural folds.
  • Secondary Structure: Predicting local protein structure (helix, strand, other).
  • Fluorescence: Predicting protein fluorescence based on sequence mutations.
  • Stability: Predicting protein stability scores.

These tasks utilize datasets curated from sources like the SCOP 1.75 database and specific experimental datasets (e.g., Deep Mutational Scan (DMS) data).

Key Insights and Ideas

  • Efficiency Matters: The paper strongly suggests that simply porting large NLP models to proteins is not the most efficient approach. CPCProt demonstrates that a significantly smaller model can achieve comparable or even better results.
  • Downstream Assessment is Tricky: Evaluating embeddings solely based on downstream task performance can be misleading. Results can vary significantly depending on the downstream model (e.g., neural network vs. linear classifier) and the evaluation metric used. This highlights the need for careful consideration of evaluation protocols.
  • Contrastive Learning Nuances: The study explores hyperparameters like batch size and the context window size (K). Interestingly, increasing the number of negative samples (proxied by batch size) did not consistently improve performance, and larger K values could even decrease performance, suggesting that more data isn't always better and the relationship between contrastive learning and performance is complex.
  • Implicit Biological Information: CPCProt can capture subtle biological properties, like protein stability, even without explicit labels for those properties during pretraining. This is demonstrated by t-SNE visualizations showing that CPCProt embeddings group proteins by stability more effectively than raw sequence embeddings.

Key Results

  • Parameter Efficiency: CPCProt uses 2% to 10% of the parameters compared to benchmarked models like BERT and UniRep, while achieving comparable performance.
  • Competitive Performance: On downstream tasks like fluorescence, CPCProt achieved higher performance metrics (e.g., higher ρ, lower MSE) than many baselines. For other tasks like secondary structure and stability, it achieved comparable performance with a fraction of the parameters.
  • Fluorescence Task: CPCProt variants showed particularly strong results on the fluorescence task, indicating its ability to capture fine-grained local sequence effects.
  • Remote Homology Nuance: While CPCProt might not perform as well as some models with a neural network head on remote homology, it outperforms them when evaluated with simpler linear regression and kNN classifiers, highlighting the impact of downstream model choice.

Limitations and Future Work

  • Negative Sampling Strategy: The paper acknowledges that the choice of negative samples for approximating the marginal distribution affects the tightness of the bound. While in-batch negative sampling was used for efficiency, a more rigorous examination of optimal negative sampling strategies for proteins is needed.
  • Hyperparameter Tuning: The study explored hyperparameters like batch size and K, but the complex relationship between contrastive learning parameters and downstream performance suggests further investigation is warranted.
  • Evaluation Protocol Consistency: The authors emphasize the difficulty in definitively evaluating embeddings due to inconsistent results across different downstream models and metrics. Future work could focus on developing more robust and consistent evaluation protocols.
  • Broader Applicability: While CPCProt shows promise for protein sequences, the authors hope their work will inform the development of similar methods for other biological sequences and modalities.
  • Model Size vs. Performance: The paper found no clear correlation between increasing the number of parameters and downstream performance, even for CPCProt variants. This suggests that architectural choices and training objectives might be more critical than sheer model size for protein representation learning.

Summary updated

Abstract

Abstract Pretrained embedding representations of biological sequences which capture meaningful properties can alleviate many problems associated with supervised learning in biology. We apply the principle of mutual information maximization between local and global information as a self-supervised pretraining signal for protein embeddings. To do so, we divide protein sequences into fixed size fragments, and train an autoregressive model to distinguish between subsequent fragments from the same protein and fragments from random proteins. Our model, CPCProt, achieves comparable performance to state-of-the-art self-supervised models for protein sequence embeddings on various downstream tasks, but reduces the number of parameters down to 2% to 10% of benchmarked models. Further, we explore how downstream assessment protocols affect embedding evaluation, and the effect of contrastive learning hyperparameters on empirical performance. We hope that these results will inform the development of contrastive learning methods in protein biology and other modalities.
BibTeX
@misc{lu2020self,
  author = {Amy X. Lu and Haoran Zhang and Marzyeh Ghassemi and Alan M. Moses},
  title = {{Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization}},
  year = {2020},
  doi = {10.1101/2020.09.04.283929},
  howpublished = {bioRxiv},
}
Figure