Abstract
We designed a fast similarity search engine for large molecular libraries: FPScreen. We downloaded 100 million molecules' structure files in PubChem with SDF extension, then applied the computational chemistry tool RDKit to convert each structure file into one line of text in MACCS format and stored them in a text file as our molecule library. The similarity search engine compares similarity while traversing the 166-bit strings in the library file line by line. FPScreen can complete a similarity search through 100 million entries in our molecule library within one hour. Additionally, we divided our library into several strides for parallel processing. FPScreen was developed in a web-based (B/S) mode.
Introduction
In the early stages of drug discovery, researchers use "virtual screening" to find molecules in a library that are structurally similar to a known active compound. As chemical databases like PubChem grow into the hundreds of millions, the computational cost of these searches has skyrocketed. Standard tools often struggle with this scale; performing a similarity search across 95 million molecules using the popular RDKit package can take over five days.
FPScreen aims to bridge this gap by providing a tool that is both "biology-fast" and hardware-efficient. It is a high-speed molecular similarity search engine capable of scanning nearly 100 million compounds in under 40 minutes on a standard desktop computer, achieving a 200x speedup over standard implementations.
Methods
FPScreen utilizes optimized data processing techniques to handle massive datasets:
- Fingerprinting: The system uses MACCS keys, a 166-bit binary string where each bit represents the presence or absence of specific structural features.
- Vectorized Search: Instead of comparing molecules one-by-one in a loop, FPScreen treats the library as a massive matrix and the target molecule as a vector. By using matrix broadcasting, it performs simultaneous calculations across the entire dataset.
- Distance Metric: Similarity is calculated using the Manhattan distance. A distance of 0.0 indicates a very high structural similarity.
- System Architecture: The tool uses a decoupled Browser/Server architecture. A Windows-based front end handles the user interface, while a Linux-based Python engine handles the heavy matrix calculations. The two systems communicate via a queuing scheme to ensure UI responsiveness.
Datasets and Parallelization
The primary dataset was the PubChem small molecule library. After filtering and processing, the final searchable library contained 95,417,114 molecules. To manage this volume, the library was divided into three "strides" (Dataset A, B, and C) for parallel processing.
Key Results
FPScreen completed a search of 95.4 million molecules in approximately 40 minutes on a modest Intel i5-6500 with 16GB of RAM. Validation against known PubChem CIDs showed that the tool accurately identifies near-neighbors. For example, searching for CID 11050016 returned CID 11092947 with a Manhattan distance of 0.0, indicating nearly identical structural features.
Conclusion and Future Work
FPScreen proves that massive-scale informatics does not necessarily require a high-performance computing (HPC) cluster if the software is sufficiently optimized via vectorization. While the 166-bit MACCS keys are efficient, they have limits in capturing subtle structural differences. Future work will focus on implementing multithreading to further reduce search times and increasing the number of features in the molecular fingerprints to improve search resolution.
Summary updated