PSSM-Distil: Protein Secondary Structure Prediction (PSSP) on Low-Quality PSSM by Knowledge Distillation with Contrastive Learning

Qin Wang, Boyuan Wang, Zhenlei Xu, Jiaxiang Wu, Peilin Zhao, Zhen Li, Sheng Wang, Junzhou Huang, Shuguang Cui

Abstract

Protein Secondary Structure Prediction (PSSP) is a cornerstone of computational biology, yet its accuracy remains heavily dependent on the availability of evolutionary information. For proteins with few known homologs, the resulting low-quality Position-Specific Scoring Matrices (PSSM) lead to a significant performance drop. We propose PSSM-Distil, a framework that addresses this "homology gap" through a cross-modal enhancement task. By employing teacher-student distillation, contrastive learning, and pre-trained protein language models (BERT), PSSM-Distil effectively "denoises" low-quality PSSMs. Our results demonstrate a substantial improvement in prediction accuracy, particularly in extreme cases with zero homologs, setting a new state-of-the-art for robust PSSP.

Introduction

Predicting a protein's local folding patterns—alpha-helices, beta-sheets, and coils—is essential for determining its overall 3D structure and biological function. Modern high-accuracy models rely on PSSMs, which act as evolutionary "fingerprints" generated through Multiple Sequence Alignment (MSA). However, this reliance creates an "Achilles' heel": the homology gap. Rare or newly discovered proteins often lack sufficient homologs, resulting in low-quality PSSMs that cause prediction accuracy to plummet from approximately 85% to 65%. While previous methods like "Bagging" attempted feature reconstruction, they often failed to capture high-level semantic distributions, leading to brittle performance in data-scarce scenarios.

Methodology

PSSM-Distil treats PSSP as a feature recovery and enhancement task. The framework consists of several integrated components designed to bridge the gap between information-rich and information-poor inputs.

  • Teacher-Student Distillation: A "Teacher" network is pre-trained on high-quality PSSMs. A "Student" network is then trained on low-quality versions of the same proteins, forced to mimic the Teacher’s output distributions. This allows the Student to learn structural patterns even from noisy inputs.
  • EnhanceNet: This module combines BiLSTMs for long-range dependencies and 1D-CNNs for local structural motifs. It transforms low-quality PSSMs into "Enhanced PSSMs."
  • Contrastive Learning (CL): To ensure semantic alignment, a triplet loss is employed. This forces the features of an enhanced PSSM to be closer to the high-quality ground truth in the latent space while pushing away unrelated or low-quality features.
  • BERT Pseudo-PSSM: For proteins with zero homologs, the model utilizes a pre-trained BERT model. By predicting masked residues, BERT provides a "pseudo-evolutionary" prior that serves as critical auxiliary information for the EnhanceNet.
The PSSM-Distil architecture, illustrating the interaction between the Enhance Net, Teacher-Student distillation, and the various loss functions (MSE, KD, and CL).

Results and Discussion

The model was evaluated on standard benchmarks (CullPDB, CB513) and a newly constructed large-scale dataset, BC40, containing nearly 37,000 non-redundant sequences. PSSM-Distil consistently outperformed previous state-of-the-art (SOTA) methods.

Key findings include:

  • Overall Gains: PSSM-Distil improves average accuracy by 6% over methods like Bagging.
  • Robustness in Data Scarcity: In "zero-homolog" cases, the improvement reaches nearly 8%. On the BC40 dataset, for sequences with zero MSA counts, PSSM-Distil achieved 73.7% accuracy, significantly higher than the 66.1% achieved by the previous best method.
  • Ablation Study: Removing the BERT component led to a 3% drop in accuracy for extreme cases, confirming that protein language models can effectively substitute for missing evolutionary data.
Accuracy improvement of PSSM-Distil compared to the Bagging method across different MSA counts and Meff score ranges, highlighting superior performance in low-diversity scenarios.

Conclusion

PSSM-Distil demonstrates that knowledge distillation and contrastive learning are powerful tools for feature recovery in bioinformatics. By aligning the semantic distributions of low- and high-quality evolutionary data, the framework successfully mitigates the homology gap. Future work will explore extending this approach to 8-state PSSP and integrating enhanced PSSM features into downstream tasks like 3D tertiary structure prediction.

Summary updated

BibTeX
@inproceedings{wang2021pssm,
  author = {Qin Wang and Boyuan Wang and Zhenlei Xu and Jiaxiang Wu and Peilin Zhao and Zhen Li and Sheng Wang and Junzhou Huang and Shuguang Cui},
  title = {{PSSM-Distil: Protein Secondary Structure Prediction (PSSP) on Low-Quality PSSM by Knowledge Distillation with Contrastive Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i1.16141},
}
Figure