TL;DR
Large language models (LLMs) struggle to accurately report factual data like cancer variant frequencies, often underestimating them. This paper introduces Query Augmented Generation (QAG), a framework that integrates LLMs with the Genomic Data Commons (GDC) to provide accurate and reliable variant statistics, even for complex queries.
What Problem Does This Paper Address?
In precision oncology, researchers frequently need to determine the frequency of specific genetic variants (like mutations or copy number changes) within patient cohorts. They often compare these frequencies to large public databases, such as the Genomic Data Commons (GDC), which houses vast amounts of cancer genomics data. While LLMs can quickly answer questions about these frequencies in natural language, they are prone to "hallucinations" – generating plausible but incorrect factual information. This is particularly problematic for numerical data like variant frequencies, where even small inaccuracies can lead to misinterpretations and flawed research conclusions. Existing methods for querying complex genomic data, like the GDC portal or API, can be time-consuming and require specialized knowledge.
Methods
The core innovation is the Query Augmented Generation (QAG) architecture. Here's how it works:
- Natural Language Query: A user poses a question in plain English (e.g., "What is the frequency of mutation X in cancer Y?").
- Intent Inference: A trained BERT classifier analyzes the query to determine the user's intent (e.g., "ssm_frequency," "cnv_loss_or_gain").
- Entity Extraction: Relevant biological entities (genes, mutations, cancer types) are identified from the query.
- API Call Builder: Based on the intent and extracted entities, QAG constructs specific API calls to query the target data source (in this case, the GDC).
- Query Third-Party Sources: QAG executes these API calls against the GDC to retrieve factual data (e.g., variant frequencies).
- LLM Prompt Construction: The original user query is combined with the retrieved factual data to create an "augmented prompt" for the LLM.
- LLM Response Generation: The LLM generates a response based on this augmented prompt.
- Final Check and Confirmation: The LLM's generated frequency is cross-referenced with the GDC data to ensure accuracy and mitigate hallucinations.
This approach leverages existing LLMs but grounds their responses in verifiable, external data.
Datasets
- Genomic Data Commons (GDC): The primary data source for variant frequencies.
- CIVIC Database: Used for curating clinical evidence statements and generating synthetic questions.
- GDC-Variant-Testset (GVT): A newly constructed evaluation dataset comprising 6011 synthetic queries across various precision oncology use cases, with ground-truth variant frequencies obtained directly from the GDC API. This dataset is crucial for evaluating baseline LLM performance and the effectiveness of QAG.
Key Insights and Ideas
- LLMs Underestimate Factual Data: Baseline LLMs (even powerful ones like GPT-40, Llama-3B, and Qwen-4B) consistently underestimate variant frequencies compared to the GDC, often with high variability. This highlights a fundamental limitation of LLMs when dealing with precise numerical data without external grounding.
- External Data is Crucial for Accuracy: The performance of LLMs on factual queries is significantly improved by augmenting their prompts with data from reliable sources like the GDC. This is true regardless of LLM size, suggesting that access to up-to-date, domain-specific knowledge is paramount.
- QAG Bridges the Gap: QAG effectively bridges the gap between the natural language understanding capabilities of LLMs and the structured, factual data in knowledgebases like the GDC, enabling accurate answers to complex biological queries.
- Modular Architecture: The QAG framework is designed to be modular, allowing it to be adapted to other knowledgebases, databases, or data meshes beyond the GDC.
Key Results
- GDC-QAG Achieves Perfect Agreement: GDC-QAG demonstrated complete concordance with the GVT dataset, meaning its variant frequencies matched the ground truth from the GDC API exactly.
- Baseline LLMs Show Significant Discrepancies: Histograms of the differences between baseline LLM responses and GVT showed broad distributions, indicating substantial underestimation and variability. For example, in the
cnv_and_ssmintent, median negative differences of around -22% to -26% were observed. - Complex Queries Handled: GDC-QAG can handle complex, multi-step queries that would typically require manual effort or advanced API knowledge on the GDC portal.
- Software Availability: The QAG architecture is deployed as a Model Context Protocol (MCP) server on Hugging Face Spaces and is publicly available on GitHub, facilitating its adoption.
Limitations and Future Work
- Disease Recognition: Current disease entity recognition relies on direct matches and NLP models, which could be enhanced with more sophisticated named entity recognition.
- Gene/Mutation Entity Recognition: This is currently limited to protein mutations and string matching.
- Query Complexity: The current implementation is primarily designed for queries involving up to two variants. Generalizing to triples and higher-order combinations is a future direction.
- Use-Case Driven: The framework is currently use-case driven, with templates for specific molecular mutational profiles. Expanding to accommodate more biological use cases and modalities (e.g., RNA data, fusions) will require retraining the BERT model and extending API integrations.
- Agentic Frameworks: The authors suggest that training open-source LLMs within agentic frameworks, using paired ground-truth datasets, could further simplify querying across modalities and lead to novel discoveries.
Summary updated