Biological Query and Pes2o Corpus Embedding Dataset

Seth Ockerman, Amal Gueroudji, Song Young Oh, Robert Underwood, Nicholas Chia, Kyle Chard, Robert Ross, Shivaram VenkataramanarXiv 2025 · DOI · arXiv

Abstract

This dataset contains the query and pes2o embeddings associated with the study "Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant," presented at the SC'25 workshop on Frontiers in Generative AI for HPC. The workload leverages vector databases to contextualize raw biological data records with information from research papers, intended for use in biological Retrieval-Augmented Generation (RAG) systems. All embeddings were generated using the Qwen3-Embedding-4B model. This paper provides an empirical evaluation of Qdrant on the Polaris supercomputer, identifying critical performance bottlenecks and scaling characteristics in high-performance computing (HPC) environments.

Introduction

Vector databases have become essential for semantic search, serving as the retrieval component in RAG workflows that allow Large Language Models (LLMs) to access vast scientific libraries. While widely deployed in cloud environments, the behavior of distributed vector databases on HPC hardware—characterized by high-speed interconnects and massive GPU clusters—remains under-explored. This study investigates whether these systems can scale linearly and identifies potential bottlenecks using a realistic biological research workload.

Methodology and Data

The evaluation was conducted using Qdrant on the Polaris supercomputer at Argonne National Laboratory. The workflow consists of four phases: embedding generation, data insertion, index building (using HNSW graphs), and querying.

Architectural approaches for distributed vector databases: stateful workers vs. separate compute and state.

Datasets

Two primary datasets were used to create the scientific RAG benchmark:

  • peS2o Corpus: Approximately 8.3 million full-text academic papers, with one embedding generated per paper.
  • BV-BRC: A subset of 22,723 genomic terms used to generate realistic queries. The resulting embedding dataset has been publicly released to facilitate further HPC-based vector search research.

Performance Bottlenecks and Insights

  • Inference Dominance: During embedding generation, model inference accounted for 98.5% of the total runtime, while I/O operations were negligible. This indicates that GPU compute capacity, rather than storage speed, is the primary bottleneck for scientific RAG preparation.
  • CPU Saturation: Index building is highly CPU-intensive. A single Qdrant worker can utilize nearly the entire CPU capacity of a node (90–97%), meaning that over-provisioning workers on a single node can lead to resource contention and degraded performance.
  • Communication Overhead: For smaller datasets (under 30GB), the overhead of network communication between distributed workers can outweigh the benefits of parallel search, sometimes resulting in slower query times compared to a single-worker setup.
  • Processing Models: The researchers found that Python’s multiprocessing outperformed asyncio for the CPU-heavy tasks involved in data preparation for insertion in high-throughput environments.

Experimental Results

The study demonstrated significant performance gains through distributed scaling, particularly for large-scale data ingestion and indexing.

Scaling performance of Qdrant on Polaris: Time to process data as a function of data size and number of workers.
  • Insertion Scaling: Scaling from 1 to 32 workers reduced the insertion time for the ~80GB dataset from over 8 hours to approximately 21.7 minutes.
  • Indexing Efficiency: The team achieved a 21.32x speedup in index construction by utilizing 32 workers.
  • Query Performance: Parallelization delivered a maximum speedup of 3.57x, though this benefit was only realized once the dataset size crossed the 30GB threshold.

Conclusion

This research highlights the potential and challenges of deploying distributed vector databases on HPC platforms. While massive speedups are achievable for data ingestion and indexing, query performance is sensitive to dataset size and communication overhead. Future work should explore GPU-accelerated indexing and "size-aware" database architectures that adaptively scale worker counts to optimize performance across varying data volumes.

Summary updated

Abstract

This dataset contains the query and pes2o embeddings associated with the paper "Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant", which was published in the SC'25 workshop Frontiers in Generative AI for HPC Science and Engineering: Foundations, Challenges, and Opportunities. All embeddings are generated using Qwen3-Embedding-4B. Workload Description Edited Excerpt from our TPC paper We consider an end-to-end workflow that leverages vector databases to contextualize raw data records with information from papers, which is intended to be used in biological RAGs ... The target workload uses ... terms related to genomes available through BV-BRC ... Each term is used to generate a query that searches the papers contained within the pes2o dataset ... for data related to the term. The intuition is that searching across a collection of research papers allows us to find data directly related to the target term, thereby providing better context for the information that would be supplied to a RAG system. The workload is exemplar and a prototype. Future work will further improve upon it. Queries The queries are generated from a 22,723 term subset of the full BV-BRC dataset. The "queries.npz" file has two keys: "data" - contains the raw embeddings. "meta" - contains integers that refer to a line of text in "query_text.csv". Note that the integers are zero-indexed, meaning the corresponding line +1's text was used to generate that embedding. "queries_v1.npz" contains the query embeddings used in the paper’s evaluation. "queries_v2.npz" contains the same queries, re-generated using the keyword argument prompt_name="query". Text Corpus We generate corpus embeddings from the full papers in the pes2o dataset, comprising 8,293,485 embeddings. A single embedding is produced for each paper. The "corpus.npz" file has two keys: "data" - contains the raw embeddings. "meta" - contains the ID corresponding to an extracted version of a paper's text stored in a JSON file internally at Argonne. If possible, we will release a way to associate it with the source text in the future. Prefered Citation @misc{ockerman2025exploringdistributedvectordatabases, title={Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant}, author={Seth Ockerman and Amal Gueroudji and Song Young Oh and Robert Underwood and Nicholas Chia and Kyle Chard and Robert Ross and Shivaram Venkataraman}, year={2025}, eprint={2509.12384}, archivePrefix={arXiv}, primaryClass={cs.DC}, url={https://arxiv.org/abs/2509.12384}, }
BibTeX
@misc{ockerman2025biological,
  author = {Seth Ockerman and Amal Gueroudji and Song Young Oh and Robert Underwood and Nicholas Chia and Kyle Chard and Robert Ross and Shivaram Venkataraman},
  title = {{Biological Query and Pes2o Corpus Embedding Dataset}},
  year = {2025},
  eprint = {2509.12384},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.12384},
  howpublished = {arXiv},
}
Figure