Query Augmented Generation (QAG) from the Genomic Data Commons for Accurate Variant Statistics

Aarti Venkat, William P. Wysocki, Michael Lukowski, Steven Song, A. Subramanyam, Zhenyu Zhang, Robert L. GrossmanbioRxiv 2025 · DOI

TL;DR

Large language models (LLMs) struggle to accurately report factual data like cancer variant frequencies, often underestimating them. This paper introduces Query Augmented Generation (QAG), a framework that integrates LLMs with the Genomic Data Commons (GDC) to provide accurate and reliable variant statistics, even for complex queries.

What Problem Does This Paper Address?

In precision oncology, researchers frequently need to determine the frequency of specific genetic variants (like mutations or copy number changes) within patient cohorts. They often compare these frequencies to large public databases, such as the Genomic Data Commons (GDC), which houses vast amounts of cancer genomics data. While LLMs can quickly answer questions about these frequencies in natural language, they are prone to "hallucinations" – generating plausible but incorrect factual information. This is particularly problematic for numerical data like variant frequencies, where even small inaccuracies can lead to misinterpretations and flawed research conclusions. Existing methods for querying complex genomic data, like the GDC portal or API, can be time-consuming and require specialized knowledge.

Methods

The core innovation is the Query Augmented Generation (QAG) architecture. Here's how it works:

  1. Natural Language Query: A user poses a question in plain English (e.g., "What is the frequency of mutation X in cancer Y?").
  2. Intent Inference: A trained BERT classifier analyzes the query to determine the user's intent (e.g., "ssm_frequency," "cnv_loss_or_gain").
  3. Entity Extraction: Relevant biological entities (genes, mutations, cancer types) are identified from the query.
  4. API Call Builder: Based on the intent and extracted entities, QAG constructs specific API calls to query the target data source (in this case, the GDC).
  5. Query Third-Party Sources: QAG executes these API calls against the GDC to retrieve factual data (e.g., variant frequencies).
  6. LLM Prompt Construction: The original user query is combined with the retrieved factual data to create an "augmented prompt" for the LLM.
  7. LLM Response Generation: The LLM generates a response based on this augmented prompt.
  8. Final Check and Confirmation: The LLM's generated frequency is cross-referenced with the GDC data to ensure accuracy and mitigate hallucinations.

This approach leverages existing LLMs but grounds their responses in verifiable, external data.

Datasets

  • Genomic Data Commons (GDC): The primary data source for variant frequencies.
  • CIVIC Database: Used for curating clinical evidence statements and generating synthetic questions.
  • GDC-Variant-Testset (GVT): A newly constructed evaluation dataset comprising 6011 synthetic queries across various precision oncology use cases, with ground-truth variant frequencies obtained directly from the GDC API. This dataset is crucial for evaluating baseline LLM performance and the effectiveness of QAG.

Key Insights and Ideas

  • LLMs Underestimate Factual Data: Baseline LLMs (even powerful ones like GPT-40, Llama-3B, and Qwen-4B) consistently underestimate variant frequencies compared to the GDC, often with high variability. This highlights a fundamental limitation of LLMs when dealing with precise numerical data without external grounding.
  • External Data is Crucial for Accuracy: The performance of LLMs on factual queries is significantly improved by augmenting their prompts with data from reliable sources like the GDC. This is true regardless of LLM size, suggesting that access to up-to-date, domain-specific knowledge is paramount.
  • QAG Bridges the Gap: QAG effectively bridges the gap between the natural language understanding capabilities of LLMs and the structured, factual data in knowledgebases like the GDC, enabling accurate answers to complex biological queries.
  • Modular Architecture: The QAG framework is designed to be modular, allowing it to be adapted to other knowledgebases, databases, or data meshes beyond the GDC.

Key Results

  • GDC-QAG Achieves Perfect Agreement: GDC-QAG demonstrated complete concordance with the GVT dataset, meaning its variant frequencies matched the ground truth from the GDC API exactly.
  • Baseline LLMs Show Significant Discrepancies: Histograms of the differences between baseline LLM responses and GVT showed broad distributions, indicating substantial underestimation and variability. For example, in the cnv_and_ssm intent, median negative differences of around -22% to -26% were observed.
  • Complex Queries Handled: GDC-QAG can handle complex, multi-step queries that would typically require manual effort or advanced API knowledge on the GDC portal.
  • Software Availability: The QAG architecture is deployed as a Model Context Protocol (MCP) server on Hugging Face Spaces and is publicly available on GitHub, facilitating its adoption.

Limitations and Future Work

  • Disease Recognition: Current disease entity recognition relies on direct matches and NLP models, which could be enhanced with more sophisticated named entity recognition.
  • Gene/Mutation Entity Recognition: This is currently limited to protein mutations and string matching.
  • Query Complexity: The current implementation is primarily designed for queries involving up to two variants. Generalizing to triples and higher-order combinations is a future direction.
  • Use-Case Driven: The framework is currently use-case driven, with templates for specific molecular mutational profiles. Expanding to accommodate more biological use cases and modalities (e.g., RNA data, fusions) will require retraining the BERT model and extending API integrations.
  • Agentic Frameworks: The authors suggest that training open-source LLMs within agentic frameworks, using paired ground-truth datasets, could further simplify querying across modalities and lead to novel discoveries.

Summary updated

Abstract

Abstract In precision oncology, researchers often use public knowledgebases to check somatic variant frequencies against their cohort data. Large language models (LLMs) can quickly answer questions on somatic variant frequencies, but often hallucinate and give inaccurate results for factual data. Using synthetic queries, we show that somatic variant frequencies in baseline LLM responses are underestimated compared to the Genomic Data Commons (GDC), the world’s largest data commons for cancer research. We present a modular architecture called Query Augmented Generation (QAG) for integrating LLMs with high-quality data from a third party data source such as a data commons, knowledgebase or database. We apply QAG to the GDC to help researchers obtain accurate frequencies for somatic variants, copy number variants, and MSI status—even for complex queries requiring multiple steps in the GDC portal and API. Our software is deployed as a model context protocol (MCP) server on Hugging Face and available on GitHub.
BibTeX
@misc{venkat2025query,
  author = {Aarti Venkat and William P. Wysocki and Michael Lukowski and Steven Song and A. Subramanyam and Zhenyu Zhang and Robert L. Grossman},
  title = {{Query Augmented Generation (QAG) from the Genomic Data Commons for Accurate Variant Statistics}},
  year = {2025},
  doi = {10.1101/2025.09.02.673797},
  howpublished = {bioRxiv},
}
Figure