TL;DR
This paper introduces a new method, Sliced-Wasserstein Embeddings (SWE), to aggregate protein language model (PLM) outputs into fixed-length representations. SWE outperforms traditional average pooling, especially for smaller PLMs and longer protein sequences, making powerful protein analysis more accessible.
What Problem Does This Paper Address?
Protein language models (PLMs) are revolutionizing biology by encoding protein sequences into rich numerical representations (embeddings). However, PLMs generate embeddings for each amino acid (residue) in a protein, resulting in variable-length outputs that depend on the protein's length. Most downstream tasks, like predicting protein function or drug interactions, require a single, fixed-length embedding for the entire protein. The common solution, average pooling, simply averages these residue-level embeddings. The problem is that this averaging might discard crucial information, especially from specific important residues, and doesn't adapt to the varying importance of different residues. This paper tackles the challenge of creating a more sophisticated and effective way to summarize these variable-length PLM outputs into a single, informative protein-level representation.
Methods
The core innovation is the Sliced-Wasserstein Embedding (SWE) method. Instead of simple averaging, SWE treats the set of residue-level embeddings from a PLM as samples from a probability distribution. It then uses principles from Optimal Transport (OT) to compare this distribution to a set of learned "reference" distributions.
Here's a breakdown:
- Conceptualization: Residue embeddings {xᵢ} are viewed as samples from a distribution.
- Reference Set: A set of
mlearnable reference embeddings {x₀ⱼ} is established. - Slicing: To make OT computationally feasible in high dimensions, the method "slices" the embedding space. This involves projecting the residue embeddings and reference embeddings onto
Ldifferent random directions (learned linear mapswₗ). This transforms the high-dimensional embeddings intoLsets of 1D distributions. - Optimal Transport on 1D Distributions: For these 1D distributions, the optimal "transportation plan" (Monge coupling) can be computed efficiently. This plan essentially describes how to move mass from one distribution to another. The paper details how to compute these couplings, handling cases where the number of residue embeddings (
n) differs from the number of reference embeddings (m). - Aggregation: The computed couplings from all
Lslices are concatenated and then combined (using another learned projectionw) to form the final fixed-length protein embedding.
The authors also propose a "SWE_Simple" variant where the reference set and slicing directions are randomly initialized and kept frozen, learning only the final combination. This reduces the number of trainable parameters significantly.
Datasets
The paper evaluates its method on several established datasets for different protein property prediction tasks:
- Drug-Target Interaction (DTI) Prediction:
- Davis: A dataset for drug-target affinity prediction.
- Binding-DB: A larger dataset for drug-target interaction prediction.
- Out-of-domain Drug-Target Affinity Prediction:
- TDC DTI Domain Generalization (DTI-DG) Benchmark: This benchmark focuses on predicting drug-target affinities for interactions patented in a later period, trained on data from an earlier period, testing generalization.
- Sub-cellular Localization (SCL): A 10-class protein property prediction task.
- Enzyme Commission (EC) Prediction: A binary classification task for predicting enzyme function.
The paper does not introduce new datasets but uses existing benchmarks to evaluate its aggregation method.
Key Insights and Ideas
- Beyond Averaging: The fundamental insight is that simple averaging of residue embeddings is suboptimal because it treats all residues equally, potentially losing critical information. SWE offers a learnable way to prioritize important residues.
- Optimal Transport for Representation Learning: Applying OT principles, which are powerful for comparing distributions, to learn protein representations is a novel and effective approach.
- Democratizing PLMs: SWE's ability to boost the performance of smaller PLMs is crucial. Smaller models require less computational power (GPU memory and time), making advanced protein analysis more accessible to researchers with limited resources.
- Length Matters: The gains from SWE are more pronounced for longer protein sequences, directly supporting the hypothesis that average pooling loses more information in these cases.
- Learnability is Key: The performance improvement from training the SWE parameters (reference set and slicers) suggests that task-specific adaptation of the aggregation layer is beneficial.
Key Results
- Superior Performance: SWE consistently outperforms average pooling across various PLMs (ESM-2 and ProGen2) and downstream tasks, particularly for binary DTI prediction.
- Smaller PLMs Benefit Most: SWE enables smaller PLMs to achieve performance comparable to or even exceeding larger PLMs aggregated with average pooling. For instance, on the Binding-DB DTI task, SWE can make smaller ProGen2 models perform as well as larger ones.
- Longer Sequences Advantage: The performance gains of SWE over average pooling increase with protein sequence length, as visualized in Figure 3.
- Competitive Benchmarking: On the TDC DTI-DG leaderboard, the proposed method achieved a strong ranking.
- Task-Specific Gains: While SWE_Simple (frozen parameters) performs well, training the SWE parameters leads to further accuracy improvements in tasks like sub-cellular localization and EC prediction.
Limitations and Future Work
- Computational Cost: While SWE is more efficient than using larger PLMs with average pooling, it still introduces some computational overhead compared to simple average pooling due to the OT calculations and learnable parameters.
- Interpretability: The authors suggest future work could focus on enhancing the interpretability of SWE, potentially by linking learned reference embeddings to biological concepts like active sites.
- Broader Applications: The authors believe their OT-based approach could be extended beyond PLMs to other biological sequence models (e.g., DNA) and foundation models.
- Auxiliary Information: Incorporating auxiliary loss functions to encode more biological intuition into the aggregation process is another promising avenue.
Summary updated