Aggregating Residue-Level Protein Language Model Embeddings with Optimal Transport

Navid NaderiAlizadeh, Rohit Singh

TL;DR

This paper introduces a new method, Sliced-Wasserstein Embeddings (SWE), to aggregate protein language model (PLM) outputs into fixed-length representations. SWE outperforms traditional average pooling, especially for smaller PLMs and longer protein sequences, making powerful protein analysis more accessible.

What Problem Does This Paper Address?

Protein language models (PLMs) are revolutionizing biology by encoding protein sequences into rich numerical representations (embeddings). However, PLMs generate embeddings for each amino acid (residue) in a protein, resulting in variable-length outputs that depend on the protein's length. Most downstream tasks, like predicting protein function or drug interactions, require a single, fixed-length embedding for the entire protein. The common solution, average pooling, simply averages these residue-level embeddings. The problem is that this averaging might discard crucial information, especially from specific important residues, and doesn't adapt to the varying importance of different residues. This paper tackles the challenge of creating a more sophisticated and effective way to summarize these variable-length PLM outputs into a single, informative protein-level representation.

Methods

The core innovation is the Sliced-Wasserstein Embedding (SWE) method. Instead of simple averaging, SWE treats the set of residue-level embeddings from a PLM as samples from a probability distribution. It then uses principles from Optimal Transport (OT) to compare this distribution to a set of learned "reference" distributions.

Here's a breakdown:

  • Conceptualization: Residue embeddings {xᵢ} are viewed as samples from a distribution.
  • Reference Set: A set of m learnable reference embeddings {x₀ⱼ} is established.
  • Slicing: To make OT computationally feasible in high dimensions, the method "slices" the embedding space. This involves projecting the residue embeddings and reference embeddings onto L different random directions (learned linear maps wₗ). This transforms the high-dimensional embeddings into L sets of 1D distributions.
  • Optimal Transport on 1D Distributions: For these 1D distributions, the optimal "transportation plan" (Monge coupling) can be computed efficiently. This plan essentially describes how to move mass from one distribution to another. The paper details how to compute these couplings, handling cases where the number of residue embeddings (n) differs from the number of reference embeddings (m).
  • Aggregation: The computed couplings from all L slices are concatenated and then combined (using another learned projection w) to form the final fixed-length protein embedding.

The authors also propose a "SWE_Simple" variant where the reference set and slicing directions are randomly initialized and kept frozen, learning only the final combination. This reduces the number of trainable parameters significantly.

Datasets

The paper evaluates its method on several established datasets for different protein property prediction tasks:

  • Drug-Target Interaction (DTI) Prediction:
    • Davis: A dataset for drug-target affinity prediction.
  • Binding-DB: A larger dataset for drug-target interaction prediction.
  • Out-of-domain Drug-Target Affinity Prediction:
    • TDC DTI Domain Generalization (DTI-DG) Benchmark: This benchmark focuses on predicting drug-target affinities for interactions patented in a later period, trained on data from an earlier period, testing generalization.
  • Sub-cellular Localization (SCL): A 10-class protein property prediction task.
  • Enzyme Commission (EC) Prediction: A binary classification task for predicting enzyme function.

The paper does not introduce new datasets but uses existing benchmarks to evaluate its aggregation method.

Key Insights and Ideas

  • Beyond Averaging: The fundamental insight is that simple averaging of residue embeddings is suboptimal because it treats all residues equally, potentially losing critical information. SWE offers a learnable way to prioritize important residues.
  • Optimal Transport for Representation Learning: Applying OT principles, which are powerful for comparing distributions, to learn protein representations is a novel and effective approach.
  • Democratizing PLMs: SWE's ability to boost the performance of smaller PLMs is crucial. Smaller models require less computational power (GPU memory and time), making advanced protein analysis more accessible to researchers with limited resources.
  • Length Matters: The gains from SWE are more pronounced for longer protein sequences, directly supporting the hypothesis that average pooling loses more information in these cases.
  • Learnability is Key: The performance improvement from training the SWE parameters (reference set and slicers) suggests that task-specific adaptation of the aggregation layer is beneficial.

Key Results

  • Superior Performance: SWE consistently outperforms average pooling across various PLMs (ESM-2 and ProGen2) and downstream tasks, particularly for binary DTI prediction.
  • Smaller PLMs Benefit Most: SWE enables smaller PLMs to achieve performance comparable to or even exceeding larger PLMs aggregated with average pooling. For instance, on the Binding-DB DTI task, SWE can make smaller ProGen2 models perform as well as larger ones.
  • Longer Sequences Advantage: The performance gains of SWE over average pooling increase with protein sequence length, as visualized in Figure 3.
  • Competitive Benchmarking: On the TDC DTI-DG leaderboard, the proposed method achieved a strong ranking.
  • Task-Specific Gains: While SWE_Simple (frozen parameters) performs well, training the SWE parameters leads to further accuracy improvements in tasks like sub-cellular localization and EC prediction.

Limitations and Future Work

  • Computational Cost: While SWE is more efficient than using larger PLMs with average pooling, it still introduces some computational overhead compared to simple average pooling due to the OT calculations and learnable parameters.
  • Interpretability: The authors suggest future work could focus on enhancing the interpretability of SWE, potentially by linking learned reference embeddings to biological concepts like active sites.
  • Broader Applications: The authors believe their OT-based approach could be extended beyond PLMs to other biological sequence models (e.g., DNA) and foundation models.
  • Auxiliary Information: Incorporating auxiliary loss functions to encode more biological intuition into the aggregation process is another promising avenue.

Summary updated

Abstract

Protein language models (PLMs) have emerged as powerful approaches for mapping protein sequences into informative embeddings suitable for a range of applications. PLMs, as well as many other protein representation schemes, generate per-token (i.e., per-residue) representations, leading to variable-sized outputs based on protein length. This variability presents a challenge for protein-level prediction tasks, which require uniform-sized embeddings for consistent analysis across different proteins. Prior work has typically resorted to average pooling to summarize token-level PLM outputs. It is, however, unclear if such an aggregation operation effectively prioritizes the relevant information across token-level representations. Addressing this, we introduce a novel method utilizing sliced-Wasserstein embeddings to convert variable-length PLM outputs into fixed-length protein-level representations. Inspired by the success of optimal transport techniques in representation learning, we first conceptualize per-token PLM outputs as samples from a probabilistic distribution. We then employ sliced-Wasserstein distances to map these samples against a learnable reference set, creating a Euclidean embedding in the output space. The resulting embedding is agnostic to the length of the input and represents the entire protein. Across a range of state-of-the-art pre-trained ESM-2 PLMs, with varying model sizes, we show the superiority of our method over average pooling for protein-drug and protein-protein interaction. Our aggregation scheme is especially effective when model size is constrained, enabling smaller-scale PLMs to match or exceed the performance of average-pooled larger-scale PLMs. Since using smaller models reduces computational resource requirements, our approach not only promises more accurate inference but can also help democratize access to foundation models.
BibTeX
@misc{naderializadeh2024aggregating,
  author = {Navid NaderiAlizadeh and Rohit Singh},
  title = {{Aggregating Residue-Level Protein Language Model Embeddings with Optimal Transport}},
  year = {2024},
  url = {https://openreview.net/forum?id=D5g9Q21uD4},
  howpublished = {OpenReview},
}
Figure