SafeBench-Seq: A Homology-Clustered, CPU-Only Baseline for Protein Hazard Screening with Physicochemical/Composition Features and Cluster-Aware Confidence Intervals

Muhammad Haris KhanarXiv 2025 · DOI · arXiv

Introduction

As generative AI (GenAI) advances, the ability to design novel proteins brings both promise and peril. A significant concern is the creation of "synthetic homologs"—hazardous proteins engineered to function like known toxins while evading traditional sequence-based biosecurity screens. SafeBench-Seq addresses this by providing a lightweight, CPU-only baseline for screening that emphasizes homology-aware evaluation.

The Challenge: Overestimating Security

Traditional screening tools like BLAST rely on high sequence similarity. However, AI-designed toxins can maintain function while appearing distinct at the sequence level. Furthermore, many machine learning models in this space are evaluated using "random splits," where similar proteins appear in both training and test sets. This leads to inflated performance metrics that fail to reflect a model's true ability to detect novel, never-before-seen hazards.

The SafeBench-Seq Methodology

The SafeBench-Seq pipeline is designed for transparency, accessibility, and rigorous evaluation:

  • Handcrafted Features: Instead of computationally expensive "black-box" embeddings, the model utilizes 20 amino acid composition frequencies and global physicochemical descriptors (e.g., molecular weight, isoelectric point, and hydropathy).
  • CPU-Only Models: The framework employs standard classifiers—Logistic Regression, Linear SVM, and Random Forest—that run efficiently on standard hardware without requiring GPUs.
  • Homology Clustering: To simulate real-world challenges, sequences are clustered at $\le 40\%$ identity using CD-HIT. Evaluation is performed using "cluster-level holdouts," ensuring the test set contains no sequences from clusters present in the training data.
  • Probability Calibration: Techniques like Isotonic Regression and Platt scaling are used to ensure predicted probabilities are reliable for high-confidence decision-making.

Dataset and Length Bias Mitigation

The study utilizes a balanced dataset of 854 sequences. Hazards (positives) are sourced from the SafeProtein benchmark, while benign sequences (negatives) are sampled from UniProtKB. A critical aspect of the dataset preparation was matching the lengths of benign and hazardous proteins to prevent the models from learning trivial length-based associations.

Figure 1: Distribution of sequence lengths for benign and harmful proteins, showing the effort to match distributions across classes.

Results: The Robustness Gap

The most significant finding is the "Robustness Gap"—the sharp decline in performance when moving from random splits to homology-clustered splits. For instance, the True Positive Rate at a 1% False Positive Rate (TPR@1%FPR) dropped from approximately 0.27 to 0.12.

Figure 6: AUROC performance across sequence length bins. The comparison between random and cluster splits highlights the performance drop when models face novel protein clusters.

Despite the difficulty of the cluster split, the Random Forest (RF) model achieved a respectable AUROC of 0.919. However, the low TPR at strict thresholds (catching only ~12% of hazards at 1% FPR) underscores the difficulty of high-volume screening without "alarm fatigue."

Conclusion and Future Directions

SafeBench-Seq provides a necessary "reality check" for protein hazard screening. While efficient and tractable, the model's reliance on sequence-only data is a limitation, as it ignores 3D structural information where function often resides. Future work will explore integrating lightweight structural proxies and biological cues, such as signal peptides, to improve detection while maintaining the tool's low-latency, CPU-only profile.

Summary updated

Abstract

Foundation models for protein design raise concrete biosecurity risks, yet the community lacks a simple, reproducible baseline for sequence-level hazard screening that is explicitly evaluated under homology control and runs on commodity CPUs. We introduce SafeBench-Seq, a metadata-only, reproducible benchmark and baseline classifier built entirely from public data (SafeProtein hazards and UniProt benigns) and interpretable features (global physicochemical descriptors and amino-acid composition). To approximate "never-before-seen" threats, we homology-cluster the combined dataset at <=40% identity and perform cluster-level holdouts (no cluster overlap between train/test). We report discrimination (AUROC/AUPRC) and screening-operating points (TPR@1% FPR; FPR@95% TPR) with 95% bootstrap confidence intervals (n=200), and we provide calibrated probabilities via CalibratedClassifierCV (isotonic for Logistic Regression / Random Forest; Platt sigmoid for Linear SVM). We quantify probability quality using Brier score, Expected Calibration Error (ECE; 15 bins), and reliability diagrams. Shortcut susceptibility is probed via composition-preserving residue shuffles and length-/composition-only ablations. Empirically, random splits substantially overestimate robustness relative to homology-clustered evaluation; calibrated linear models exhibit comparatively good calibration, while tree ensembles retain slightly higher Brier/ECE. SafeBench-Seq is CPU-only, reproducible, and releases metadata only (accessions, cluster IDs, split labels), enabling rigorous evaluation without distributing hazardous sequences.
BibTeX
@misc{khan2025safebench,
  author = {Muhammad Haris Khan},
  title = {{SafeBench-Seq: A Homology-Clustered, CPU-Only Baseline for Protein Hazard Screening with Physicochemical/Composition Features and Cluster-Aware Confidence Intervals}},
  year = {2025},
  eprint = {2512.17527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.17527},
}
Figure