Discriminative protein sequence modelling with Latent Space Diffusion

Eoin Quinn, Ghassene Jebali, Maxime Seince, Oliver BentarXiv 2025 · DOI · arXiv

Introduction

Protein sequence modeling has traditionally relied on discrete representations, treating amino acids as tokens in a language. While Masked Language Models (MLMs) like ESM have achieved significant success by learning the "grammar" of proteins, they are primarily reconstructive. Researchers at InstaDeep have recently proposed a shift towards Latent Space Diffusion (LSD), exploring whether mapping protein sequences into a continuous latent space can yield more powerful representations for downstream biological tasks.

The LSD Architecture

The proposed framework decouples representation learning into two distinct phases: manifold learning and distributional modeling.

  1. Protein Sequence Autoencoder (AE): This component maps discrete amino acid tokens into a continuous latent vector $z$. To ensure this space is informative, the authors introduced novel bottleneck strategies like Token Norm (LSD-TN) and Noise Masking (LSD-NM).
  2. Latent Diffusion Model: A diffusion model is then trained on the fixed latent space $z$. By learning to reverse a Gaussian noise process, the model captures the underlying distribution of protein sequences, effectively learning a "score function" that represents the evolutionary forces acting on the data.
Overview of the Latent Space Diffusion (LSD) architecture, showing the autoencoder mapping to latent space $z$ (a) and the subsequent diffusion process (b).

Novel Bottleneck Strategies

A key challenge in latent diffusion for discrete data is preventing the latent space from simply memorizing the discrete tokens. The authors propose two solutions:

  • Token Norm (LSD-TN): This strategy constrains amino acids of the same type to follow the same normal distribution in the latent space, creating a structured bottleneck that encourages the model to learn categorical relationships.
  • Noise Masking (LSD-NM): This acts as a continuous-space analogue to MLM. During training, varying levels of Gaussian noise are added to the latent vectors, forcing the model to learn robust denoising features.

Experimental Evaluation

The models were pre-trained on the Uniref50 dataset and evaluated on a suite of discriminative benchmarks, including Thermostability, Human Protein-Protein Interaction (HumanPPI), Metal Ion Binding, and DeepLoc (subcellular localization).

A unique property of the LSD approach is that it provides a one-parameter family of representations indexed by the diffusion time $t$. By varying $t$, one can extract features ranging from raw latent embeddings ($t=0$) to more "denoised" distributional scores.

Performance of LSD models across various benchmarks as a function of the diffusion time parameter $t$.

Key Findings

  • Diffusion Improvements: LSD models significantly outperformed standard diffusion baselines. For instance, LSD-NM achieved a Spearman’s $\rho$ of 0.581 on the Thermostability task, compared to 0.543 for a baseline diffusion model trained on MLM embeddings.
  • The "Optimal t" Phenomenon: For certain tasks, the most discriminative representation was found at a non-zero $t$. In the HumanPPI task, performance peaked at $t \approx 0.15$, suggesting that the diffusion model's ability to capture broader distributional information can filter out local noise inherent in raw sequence data.
  • The MLM Gap: Despite these advancements, raw MLM encoders still generally outperformed diffusion-based features. This indicates that while diffusion helps organize the latent space, it does not yet fully capture the rich contextual information provided by the MLM objective.

Conclusion and Future Directions

Latent Space Diffusion represents a promising new direction for protein modeling, successfully bridging the gap between discrete sequences and continuous generative processes. While MLM remains the state-of-the-art for discriminative tasks, LSD offers a unique distributional perspective. Future work will focus on scaling these models to billions of parameters, evaluating their generative capabilities for protein design, and incorporating 3D structural data into the latent space.

Summary updated

Abstract

We explore a framework for protein sequence representation learning that decomposes the task between manifold learning and distributional modelling. Specifically we present a Latent Space Diffusion architecture which combines a protein sequence autoencoder with a denoising diffusion model operating on its latent space. We obtain a one-parameter family of learned representations from the diffusion model, along with the autoencoder's latent representation. We propose and evaluate two autoencoder architectures: a homogeneous model forcing amino acids of the same type to be identically distributed in the latent space, and an inhomogeneous model employing a noise-based variant of masking. As a baseline we take a latent space learned by masked language modelling, and evaluate discriminative capability on a range of protein property prediction tasks. Our finding is twofold: the diffusion models trained on both our proposed variants display higher discriminative power than the one trained on the masked language model baseline, none of the diffusion representations achieve the performance of the masked language model embeddings themselves.
BibTeX
@misc{quinn2025discriminative,
  author = {Eoin Quinn and Ghassene Jebali and Maxime Seince and Oliver Bent},
  title = {{Discriminative protein sequence modelling with Latent Space Diffusion}},
  year = {2025},
  eprint = {2503.18551},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.18551},
}
Figure