Introduction
As generative AI (GenAI) advances, the ability to design novel proteins brings both promise and peril. A significant concern is the creation of "synthetic homologs"—hazardous proteins engineered to function like known toxins while evading traditional sequence-based biosecurity screens. SafeBench-Seq addresses this by providing a lightweight, CPU-only baseline for screening that emphasizes homology-aware evaluation.
The Challenge: Overestimating Security
Traditional screening tools like BLAST rely on high sequence similarity. However, AI-designed toxins can maintain function while appearing distinct at the sequence level. Furthermore, many machine learning models in this space are evaluated using "random splits," where similar proteins appear in both training and test sets. This leads to inflated performance metrics that fail to reflect a model's true ability to detect novel, never-before-seen hazards.
The SafeBench-Seq Methodology
The SafeBench-Seq pipeline is designed for transparency, accessibility, and rigorous evaluation:
- Handcrafted Features: Instead of computationally expensive "black-box" embeddings, the model utilizes 20 amino acid composition frequencies and global physicochemical descriptors (e.g., molecular weight, isoelectric point, and hydropathy).
- CPU-Only Models: The framework employs standard classifiers—Logistic Regression, Linear SVM, and Random Forest—that run efficiently on standard hardware without requiring GPUs.
- Homology Clustering: To simulate real-world challenges, sequences are clustered at $\le 40\%$ identity using CD-HIT. Evaluation is performed using "cluster-level holdouts," ensuring the test set contains no sequences from clusters present in the training data.
- Probability Calibration: Techniques like Isotonic Regression and Platt scaling are used to ensure predicted probabilities are reliable for high-confidence decision-making.
Dataset and Length Bias Mitigation
The study utilizes a balanced dataset of 854 sequences. Hazards (positives) are sourced from the SafeProtein benchmark, while benign sequences (negatives) are sampled from UniProtKB. A critical aspect of the dataset preparation was matching the lengths of benign and hazardous proteins to prevent the models from learning trivial length-based associations.
Results: The Robustness Gap
The most significant finding is the "Robustness Gap"—the sharp decline in performance when moving from random splits to homology-clustered splits. For instance, the True Positive Rate at a 1% False Positive Rate (TPR@1%FPR) dropped from approximately 0.27 to 0.12.
Despite the difficulty of the cluster split, the Random Forest (RF) model achieved a respectable AUROC of 0.919. However, the low TPR at strict thresholds (catching only ~12% of hazards at 1% FPR) underscores the difficulty of high-volume screening without "alarm fatigue."
Conclusion and Future Directions
SafeBench-Seq provides a necessary "reality check" for protein hazard screening. While efficient and tractable, the model's reliance on sequence-only data is a limitation, as it ignores 3D structural information where function often resides. Future work will explore integrating lightweight structural proxies and biological cues, such as signal peptides, to improve detection while maintaining the tool's low-latency, CPU-only profile.
Summary updated