Abstract
Pretrained protein structure models without labels are crucial foundations for the majority of protein downstream applications. Conventional structure pretraining methods often follow natural language processing techniques, such as denoised reconstruction and masked language modeling, which can inadvertently destroy the spatial representation of structures. Other common methods rely on restricted supervised learning, limiting their generality. In this work, we introduce a novel unsupervised protein structure representation pretraining framework that leverages a robust protein language model (PLM). Specifically, we propose to guide structure model learning through unsupervised contrastive alignment with an existing pretrained language model. Additionally, a self-supervised structure constraint is introduced to capture intrinsic structural information. With lightweight training data, the resulting model demonstrates superior generalization. We evaluate the framework through internal tasks (e.g., contact map prediction) and external downstream tasks (e.g., protein design), demonstrating the superiority of this sequence-structure transformation approach.
Introduction
In proteomics, a significant "information asymmetry" exists: while hundreds of millions of protein sequences are available, only a tiny fraction of 3D structures have been experimentally determined. While Protein Language Models (PLMs) like ESM-2 have become highly proficient at representing sequences by capturing evolutionary information, structural encoders have historically lagged.
Traditional self-supervised techniques, such as masking, often fail for 3D structures because they can disrupt the delicate geometry required for biological function. This paper addresses the challenge by asking: can we use the rich evolutionary knowledge already embedded in sequence models to teach a structure model how to "see" proteins? The proposed framework, Cross-modal Contrastive Protein Learning (CCPL), treats protein representation as a cross-modal retrieval task, effectively bridging the gap between 1D sequences and 3D structures.
The CCPL Framework
CCPL utilizes a dual-encoder architecture to align sequence and structure modalities. The system consists of two primary components:
- Sequence Encoder: A fixed, pretrained protein language model (e.g., ESM-2) that provides high-dimensional embeddings of amino acid sequences.
- Structure Encoder: A Geometric Structure Module designed to process 3D coordinates while maintaining geometric integrity.
During the pretraining phase, the model employs Contrastive Alignment. It processes batches of paired sequences and structures, learning to maximize the similarity between a structure and its corresponding sequence while minimizing similarity with others in the batch. This alignment occurs at two levels of granularity: residue-level (fine-grained) and protein-level (coarse-grained). To prevent the structure encoder from losing physical realism during alignment, an auxiliary structural flow uses contact map prediction as a self-supervised constraint, ensuring the model retains a high-fidelity understanding of spatial distances.
Geometric Structure Module
A critical component of the framework is the structure encoder's ability to handle 3D data without being sensitive to arbitrary rotations or translations. The model utilizes Geometric Vector Perceptron (GVP) layers to achieve this.
As shown in the architecture, the GVP layers extract both scalar and vector features from the protein backbone (N, Cα, and C atoms). By utilizing local frame features, the model generates rotation-invariant representations. This ensures that the protein's "identity" remains consistent regardless of its orientation in 3D space, a prerequisite for accurate functional prediction and design.
Experimental Evaluation
The framework was pretrained using the Protein Data Bank (PDB) and augmented with predicted structures from AlphaFoldDB. Evaluation spanned several critical biological tasks:
- Protein Design (Inverse Folding): On the Ts500 benchmark, CCPL achieved a 60.3% sequence recovery rate, outperforming established models like ProteinMPNN (58.08%).
- Zero-shot Fitness Prediction: On the ProteinGym dataset, which measures the impact of mutations, CCPL achieved a Spearman rank correlation of 0.430, surpassing ESM-IF (0.422).
- Function Prediction: The model showed superior performance in Gene Ontology (GO) and Enzyme Commission (EC) tasks, demonstrating its ability to capture functional nuances that 1D-only or 3D-only models might miss.
- Fold Recognition: CCPL significantly improved the ability to identify protein folds (16.8% P@1) compared to standard GVP (10.1%) or ESM-2 (11.5%).
Conclusion
CCPL demonstrates that the evolutionary wisdom stored in protein sequences can be effectively distilled into structural models through contrastive learning. By avoiding the distortion inherent in masked modeling and leveraging the strengths of existing PLMs, CCPL provides a robust, lightweight foundation for protein structure representation. Future work will focus on scaling these models alongside advancing language models and improving data efficiency for even broader biological applications.
Summary updated