HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights

Ozan Gökdemir, Carlo Siebenschuh, Alexander Brace, Azton Wells, Brian Hsu, Kyle Hippe, Priyanka Setty, Aswathy Ajith, J. Gregory Pauloski, Varuni Sastry, Sam Foreman, Huihuo Zheng, Heng Ma, Bharat Kale, Nicholas Chia, Thomas Gibbs, Michael E. Papka, Thomas S. Brettin, Francis J. Alexander, Anima Anandkumar, Ian T. Foster, Rick Stevens, Venkatram Vishwanath, Arvind Ramanathan

TL;DR

The paper introduces HiPerRAG, a high-performance system designed to tackle the challenge of extracting and synthesizing knowledge from millions of scientific articles using Retrieval-Augmented Generation (RAG). By combining a novel document parsing method (Oreo) and a specialized encoder fine-tuning technique (ColTrast), HiPerRAG significantly improves the accuracy and efficiency of RAG for scientific Q&A, outperforming existing models and paving the way for more effective scientific discovery.

What Problem Does This Paper Address?

The sheer volume of scientific literature is exploding, making it increasingly difficult for researchers to keep up with new discoveries, avoid duplicating efforts, and foster cross-disciplinary collaboration. While Large Language Models (LLMs) offer potential for processing this information, they often "hallucinate" or generate factually incorrect content. Retrieval-Augmented Generation (RAG) aims to mitigate this by grounding LLM outputs in retrieved information. However, scaling RAG to handle millions of scientific documents presents significant hurdles: the high computational cost of parsing complex PDF documents and embedding scientific knowledge, and the difficulty of accurately aligning these embeddings with the nuanced language of science. This paper addresses these challenges by developing a scalable and accurate RAG framework specifically for scientific literature.

Methods

HiPerRAG is built upon several key innovations:

  • Oreo (Optical Recognition with Eclectic Output): This is a novel, high-throughput multimodal document parser designed to handle the complex layouts of scientific PDFs. Unlike traditional methods that rely on embedded text layers or struggle with diverse content, Oreo uses a two-stage process: first, a YOLO architecture for layout detection (identifying text, figures, tables), and then a Vision Transformer (ViT) based approach (Texify) to extract text. It supports parsing of text, figures, and tables, and significantly improves throughput and computational efficiency compared to existing parsers.
  • ColTrast: This is a query-aware encoder fine-tuning algorithm that enhances retrieval accuracy for scientific content. It combines two techniques:
  • Late Interaction: Similar to ColBERT, it compares query and document tokens for finer-grained semantic alignment, but to improve scalability, it uses pooled embeddings instead of token-by-token comparisons.
  • Contrastive Learning: This helps the model learn to distinguish between relevant and irrelevant document chunks by pulling similar embeddings closer and pushing dissimilar ones apart. The "ColTrast" loss function combines these two approaches to create embeddings that are both accurate for scientific queries and scalable.

  • HPC Infrastructure: The entire HiPerRAG workflow (parsing, chunking, encoding, retrieval) is distributed across High-Performance Computing (HPC) systems using the Parsl framework. This allows for scaling to millions of documents and thousands of GPUs. A "warmstart" optimization using a model registry is employed to minimize model loading times between tasks.

Datasets

The paper introduces two new biomedical Q/A benchmarks:

  • ProteinInteractionQA: Contains 7,591 question-answer pairs focused on protein interactions, curated from databases of antimicrobial peptides.
  • ProteinFunctionQA: Features 17,646 question-answer pairs related to the functions of antimicrobial peptides.

Additionally, they created a synthetic dataset called BioSynthQP (1,500 passages) designed to specifically evaluate the retrieval accuracy of encoders. The ColTrast finetuning dataset comprises 82,750 papers across three scientific domains: peptides, cancer, and COVID-19.

Key Insights and Ideas

  • The "Information Overload" is a Real Bottleneck: The paper strongly emphasizes that the sheer volume of scientific literature is a major impediment to scientific progress, leading to duplicated efforts and missed connections.
  • Domain-Specific RAG is Crucial: General-purpose LLMs struggle with the nuances of scientific language and factuality. HiPerRAG demonstrates that tailoring RAG components (parsing and encoding) to scientific literature yields significant improvements.
  • Scalability and Accuracy Can Coexist: The core challenge addressed is achieving both high accuracy and massive scalability. HiPerRAG's innovations in Oreo and ColTrast show that this is achievable through careful algorithmic design and HPC utilization.
  • Query-Awareness is Key for Retrieval: The ColTrast method's ability to fine-tune encoders based on specific query semantics is vital for retrieving the most relevant information from a vast corpus.
  • HPC is Not Just for Training, But for Inference at Scale: The paper highlights how HPC infrastructure is essential for deploying RAG systems that can handle millions of documents in a practical timeframe.

Key Results

  • Oreo Performance: Achieved a 4.5x speedup over the state-of-the-art Marker parser, with a 94.6x improvement in FLOP utilization. It also demonstrated superior multimodal parsing capabilities, handling figures, tables, and equations, unlike some competitors.
  • ColTrast Retrieval Accuracy: Outperformed models using only contrastive or late-interaction loss. When combined with a large scientific corpus, it significantly boosted Q/A performance.
  • Overall HiPerRAG Performance:
    • Achieved 90% accuracy on the SciQ benchmark, outperforming GPT-4.
  • Achieved 76% accuracy on PubMedQA, outperforming domain-specific models like PubMedGPT.
  • Demonstrated robust performance on the newly introduced ProteinInteractionQA and ProteinFunctionQA benchmarks.
  • Scalability: The system was successfully scaled to thousands of GPUs on supercomputers like Polaris, Sunspot, and Frontier, processing millions of documents.

Limitations and Future Work

  • Multimodality: While Oreo can parse multimodal data (images, tables), the current evaluation primarily focused on text retrieval. Future work aims to incorporate multimodal LLMs to handle images, video, and audio more comprehensively.
  • Evaluation Benchmarks: The paper acknowledges the need for more established multimodal benchmarks for scientific literature.
  • Temporal Evolution: The authors plan to explore temporal evolution strategies to predict emerging concepts or research themes.
  • Knowledge Graph Integration: The learned embeddings could be used to build ontologies and connect scientific concepts, potentially leading to new hypothesis generation.

Summary updated

Abstract

The volume of scientific literature is growing exponentially, leading to underutilized discoveries, duplicated efforts, and limited cross-disciplinary collaboration. Retrieval-Augmented Generation (RAG) offers a way to assist scientists by improving the factuality of Large Language Models (LLMs) in processing this influx of information. However, scaling RAG to handle millions of articles introduces significant challenges, including the high computational costs associated with parsing documents and embedding scientific knowledge, as well as the algorithmic complexity of aligning these representations with the nuanced semantics of scientific content. To address these issues, we introduce HiPerRAG, a RAG workflow powered by high performance computing (HPC) to index and retrieve knowledge from more than 3.6 million scientific articles. At its core are Oreo, a high-throughput model for multimodal document parsing, and ColTrast, a query-aware encoder fine-tuning algorithm that enhances retrieval accuracy by using contrastive learning and late-interaction techniques. HiPerRAG delivers robust performance on existing scientific question answering (Q/A) benchmarks and two new benchmarks introduced in this work, achieving 90% accuracy on SciQ and 76% on PubMedQA—outperforming both domain-specific models like PubMedGPT and commercial LLMs such as GPT-4. Scaling to thousands of GPUs on the Polaris, Sunspot, and Frontier supercomputers, HiPerRAG delivers million document-scale RAG workflows for unifying scientific knowledge and fostering interdisciplinary innovation.
BibTeX
@inproceedings{gokdemir2025hiperrag,
  author = {Ozan Gökdemir and Carlo Siebenschuh and Alexander Brace and Azton Wells and Brian Hsu and Kyle Hippe and Priyanka Setty and Aswathy Ajith and J. Gregory Pauloski and Varuni Sastry and Sam Foreman and Huihuo Zheng and Heng Ma and Bharat Kale and Nicholas Chia and Thomas Gibbs and Michael E. Papka and Thomas S. Brettin and Francis J. Alexander and Anima Anandkumar and Ian T. Foster and Rick Stevens and Venkatram Vishwanath and Arvind Ramanathan},
  title = {{HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights}},
  booktitle = {PASC},
  year = {2025},
  doi = {10.1145/3732775.3733586},
}
Figure