Abstract
This dataset contains the query and pes2o embeddings associated with the study "Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant," presented at the SC'25 workshop on Frontiers in Generative AI for HPC. The workload leverages vector databases to contextualize raw biological data records with information from research papers, intended for use in biological Retrieval-Augmented Generation (RAG) systems. All embeddings were generated using the Qwen3-Embedding-4B model. This paper provides an empirical evaluation of Qdrant on the Polaris supercomputer, identifying critical performance bottlenecks and scaling characteristics in high-performance computing (HPC) environments.
Introduction
Vector databases have become essential for semantic search, serving as the retrieval component in RAG workflows that allow Large Language Models (LLMs) to access vast scientific libraries. While widely deployed in cloud environments, the behavior of distributed vector databases on HPC hardware—characterized by high-speed interconnects and massive GPU clusters—remains under-explored. This study investigates whether these systems can scale linearly and identifies potential bottlenecks using a realistic biological research workload.
Methodology and Data
The evaluation was conducted using Qdrant on the Polaris supercomputer at Argonne National Laboratory. The workflow consists of four phases: embedding generation, data insertion, index building (using HNSW graphs), and querying.
Datasets
Two primary datasets were used to create the scientific RAG benchmark:
- peS2o Corpus: Approximately 8.3 million full-text academic papers, with one embedding generated per paper.
- BV-BRC: A subset of 22,723 genomic terms used to generate realistic queries. The resulting embedding dataset has been publicly released to facilitate further HPC-based vector search research.
Performance Bottlenecks and Insights
- Inference Dominance: During embedding generation, model inference accounted for 98.5% of the total runtime, while I/O operations were negligible. This indicates that GPU compute capacity, rather than storage speed, is the primary bottleneck for scientific RAG preparation.
- CPU Saturation: Index building is highly CPU-intensive. A single Qdrant worker can utilize nearly the entire CPU capacity of a node (90–97%), meaning that over-provisioning workers on a single node can lead to resource contention and degraded performance.
- Communication Overhead: For smaller datasets (under 30GB), the overhead of network communication between distributed workers can outweigh the benefits of parallel search, sometimes resulting in slower query times compared to a single-worker setup.
- Processing Models: The researchers found that Python’s
multiprocessingoutperformedasynciofor the CPU-heavy tasks involved in data preparation for insertion in high-throughput environments.
Experimental Results
The study demonstrated significant performance gains through distributed scaling, particularly for large-scale data ingestion and indexing.
- Insertion Scaling: Scaling from 1 to 32 workers reduced the insertion time for the ~80GB dataset from over 8 hours to approximately 21.7 minutes.
- Indexing Efficiency: The team achieved a 21.32x speedup in index construction by utilizing 32 workers.
- Query Performance: Parallelization delivered a maximum speedup of 3.57x, though this benefit was only realized once the dataset size crossed the 30GB threshold.
Conclusion
This research highlights the potential and challenges of deploying distributed vector databases on HPC platforms. While massive speedups are achievable for data ingestion and indexing, query performance is sensitive to dataset size and communication overhead. Future work should explore GPU-accelerated indexing and "size-aware" database architectures that adaptively scale worker counts to optimize performance across varying data volumes.
Summary updated