HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights

Ozan Gökdemir, Carlo Siebenschuh, Alexander Brace, Azton Wells, Brian Hsu, Kyle Hippe, Priyanka Vasanthakumari, Aswathy Ajith, J. Gregory Pauloski, Varuni Katti Sastry, Sam Foreman, Huihuo Zheng, Heng Ma, Bharat Kale, Nicholas Chia, Thomas Gibbs, Michael E. Papka, Thomas S. Brettin, Francis J. Alexander, Anima Anandkumar, Ian T. Foster, Rick Stevens, Venkatram Vishwanath, Arvind Ramanathan

TL;DR

The paper introduces HiPerRAG, a high-performance system for Retrieval Augmented Generation (RAG) designed to tackle the overwhelming volume of scientific literature. By combining a novel document parser (Oreo) and a specialized retrieval encoder (ColTrast) with High-Performance Computing (HPC), HiPerRAG enables efficient and accurate extraction of scientific insights from millions of documents, outperforming existing models on key benchmarks.

What Problem Does This Paper Address?

The sheer volume of scientific publications is growing at an unprecedented rate, making it nearly impossible for researchers to keep up. This "information overload" leads to duplicated efforts, missed discoveries, and limited cross-disciplinary collaboration. While Large Language Models (LLMs) offer potential solutions for processing this information, they often struggle with factual accuracy and "hallucinate" incorrect information, especially when dealing with specialized scientific content. Retrieval Augmented Generation (RAG) aims to mitigate this by grounding LLM outputs in retrieved factual information. However, scaling RAG to handle millions of scientific articles presents significant challenges: the high computational cost of parsing and embedding complex scientific documents, and the difficulty of accurately aligning semantic representations for nuanced scientific queries.

Methods

HiPerRAG addresses these challenges through several key innovations:

  • Oreo (Optical Recognition with Eclectic Output): A novel, high-throughput multimodal document parser for scientific PDFs. Unlike traditional parsers, Oreo uses a two-stage approach: first, a YOLO-based layout detection to identify text, figures, and tables, and then a Vision Transformer (ViT) with Texify to extract text. This allows it to handle complex layouts and extract information beyond just text, achieving significant speedups and comparable accuracy to state-of-the-art methods.
  • ColTrast: A query-aware encoder fine-tuning algorithm. To overcome the limitations of general-purpose LLMs in understanding scientific text, ColTrast combines two techniques:
  • Contrastive Learning: Helps the model distinguish between relevant and irrelevant document passages.
  • Late-Interaction: A more computationally intensive but precise method that compares tokens between queries and documents. ColTrast uses a novel "ColTrast loss" that balances these two approaches, optimizing for both accuracy and scalability. It also leverages pooled embeddings for similarity search, improving efficiency over token-by-token comparisons.

  • HPC Integration: The entire HiPerRAG workflow (parsing, chunking, encoding, retrieval) is distributed across High-Performance Computing (HPC) systems using the Parsl framework. This allows for massive scaling to millions of documents. A "warmstart optimization" using a model registry further improves efficiency by caching loaded models.

Datasets

  • ColTrast Finetuning Dataset: A custom dataset of 82,750 scientific papers across domains like peptides, cancer, and COVID-19. This dataset was used to fine-tune the ColTrast encoder. It includes approximately 455,894 generated question-chunk pairs.
  • ProteinInteractionQA: A new benchmark dataset with 7,591 question-answer pairs focused on protein interactions, curated from existing peptide databases.
  • ProteinFunctionQA: Another new benchmark dataset with 17,646 question-answer pairs focusing on protein functions, derived from UniProt functional descriptions.
  • BioSynthQP: A synthetic dataset of 1,500 domain-specific scientific passages designed to evaluate retrieval accuracy, featuring questions with paragraphs of varying relevance.
  • Existing Benchmarks: SciQ, PubMedQA, and LitQA were also used for evaluation.

Key Insights and Ideas

  • The "Information Overload" in Science is a Critical Bottleneck: The paper strongly emphasizes that the sheer volume of scientific literature is a major impediment to scientific progress, leading to inefficiency and missed opportunities.
  • General LLMs Struggle with Scientific Nuance: Standard LLMs, trained on broad datasets, are not optimized for the specialized language and complex relationships found in scientific literature, leading to suboptimal retrieval and factual inaccuracies.
  • Combining Contrastive Learning and Late Interaction is Key: The ColTrast loss function demonstrates that a hybrid approach, balancing the broad discrimination of contrastive learning with the fine-grained precision of late interaction, is crucial for accurate scientific retrieval.
  • HPC is Essential for Scaling Scientific AI: The paper highlights that tackling the scale of scientific literature requires leveraging massive computational resources. HiPerRAG's design is fundamentally built around HPC for efficient processing.
  • Novel Benchmarks are Needed for Scientific RAG: The introduction of ProteinInteractionQA and ProteinFunctionQA underscores the need for specialized evaluation datasets that capture the unique challenges of scientific question answering.

Key Results

  • Oreo Performance: Achieved a 4.5x speedup over state-of-the-art parsers and a 94.6x improvement in FLOP utilization. It also demonstrated superior multimodal capabilities, parsing text, figures, and tables.
  • ColTrast Performance: Outperformed baseline models on the BioSynthQP dataset, achieving higher P@10 and MRR@10 scores, especially when using the combined ColTrast loss.
  • HiPerRAG on Q/A Benchmarks:
    • Achieved 90% accuracy on SciQ using the Mixtral8x7B generator with the ColTrast-B-S encoder.
  • Achieved 76% accuracy on PubMedQA with Mixtral8x7B and PubMedBERT, outperforming both domain-specific models like PubMedGPT and commercial LLMs like GPT-4.
  • Scalability: Demonstrated strong linear scaling of parsing and semantic chunking workflows across thousands of GPUs on Polaris, Sunspot, and Frontier supercomputers.

Limitations and Future Work

  • Multimodality: While Oreo can parse multimodal data (figures, tables), the current evaluation primarily focused on text retrieval. Future work aims to incorporate multimodal LLMs for more comprehensive scientific insight extraction.
  • Temporal Evolution: The authors plan to explore temporal evolution strategies to predict emerging concepts and research trends.
  • Knowledge Graph Integration: The learned embeddings from HiPerRAG could be used to build ontologies and connect scientific concepts, potentially leading to new hypothesis generation.
  • Fine-tuning Generator Models: While HiPerRAG focuses on retrieval, future work could involve fine-tuning the generator LLMs themselves for even better performance.

Summary updated

Abstract

The volume of scientific literature is growing exponentially, leading to underutilized discoveries, duplicated efforts, and limited cross-disciplinary collaboration. Retrieval Augmented Generation (RAG) offers a way to assist scientists by improving the factuality of Large Language Models (LLMs) in processing this influx of information. However, scaling RAG to handle millions of articles introduces significant challenges, including the high computational costs associated with parsing documents and embedding scientific knowledge, as well as the algorithmic complexity of aligning these representations with the nuanced semantics of scientific content. To address these issues, we introduce HiPerRAG, a RAG workflow powered by high performance computing (HPC) to index and retrieve knowledge from more than 3.6 million scientific articles. At its core are Oreo, a high-throughput model for multimodal document parsing, and ColTrast, a query-aware encoder fine-tuning algorithm that enhances retrieval accuracy by using contrastive learning and late-interaction techniques. HiPerRAG delivers robust performance on existing scientific question answering benchmarks and two new benchmarks introduced in this work, achieving 90% accuracy on SciQ and 76% on PubMedQA-outperforming both domain-specific models like PubMedGPT and commercial LLMs such as GPT-4. Scaling to thousands of GPUs on the Polaris, Sunspot, and Frontier supercomputers, HiPerRAG delivers million document-scale RAG workflows for unifying scientific knowledge and fostering interdisciplinary innovation.
BibTeX
@inproceedings{gokdemir2025hiperrag,
  author = {Ozan Gökdemir and Carlo Siebenschuh and Alexander Brace and Azton Wells and Brian Hsu and Kyle Hippe and Priyanka Vasanthakumari and Aswathy Ajith and J. Gregory Pauloski and Varuni Katti Sastry and Sam Foreman and Huihuo Zheng and Heng Ma and Bharat Kale and Nicholas Chia and Thomas Gibbs and Michael E. Papka and Thomas S. Brettin and Francis J. Alexander and Anima Anandkumar and Ian T. Foster and Rick Stevens and Venkatram Vishwanath and Arvind Ramanathan},
  title = {{HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights}},
  booktitle = {PASC},
  year = {2025},
  doi = {10.1145/3732775.3733586},
}
Figure