ProveRAG: Provenance-Driven Vulnerability Analysis With Automated Retrieval-Augmented LLMs

Reza Fayyazi, Stella Hoyos Trueba, Michael Zuzak, Shanchieh Jay Yang

TL;DR

Cybersecurity analysts are drowning in a sea of newly discovered vulnerabilities. This paper introduces ProveRAG, a system that uses Large Language Models (LLMs) augmented with automated web data retrieval and a self-critique mechanism to analyze these vulnerabilities. ProveRAG provides verifiable evidence for its findings, significantly improving accuracy and trustworthiness in a rapidly evolving threat landscape.

What Problem Does This Paper Address?

The sheer volume of newly discovered vulnerabilities (over 300,000 since 1999, with 40,000 in 2024 alone) overwhelms cybersecurity analysts. While LLMs can help, they suffer from "hallucinations" (generating incorrect information), "temporal knowledge" limitations (struggling with data beyond their training cutoff), and "limited context windows" (inability to process large amounts of information). These issues are critical in cybersecurity, where accuracy and up-to-date information are paramount. Existing manual analysis methods are slow, and current LLM-based approaches often lack verifiable evidence, making it hard for analysts to trust their outputs.

Methods

ProveRAG employs a two-stage methodology: Generation and Evaluation.

  1. Generation Module: This module mimics how a human analyst works. It starts by retrieving information about a specific CVE (Common Vulnerabilities and Exposures) from authoritative sources like the National Vulnerability Database (NVD) and the Common Weakness Enumeration (CWE). It then uses a novel summarizing retrieval technique to condense this information, rather than simply chunking it, which helps manage large datasets and context window limitations. This summarized information is then fed to a Generative LLM (Gen. LLM) to produce detailed exploitation and mitigation strategies.
  2. Evaluation Module: This is where ProveRAG's self-critique and provenance capabilities shine. The Gen. LLM's response is evaluated by another LLM (Eval. LLM) against the retrieved evidence. The system identifies:
  3. True Positives (TPs): Accurate predictions.
  4. False Positives (FPs): Detected hallucinations.
  5. False Negatives (FNs): Failed retrievals (omitted information). Crucially, ProveRAG provides provenance by showing the exact text snippets from the original sources that support its claims, along with a rationale for its evaluation. This ensures transparency and allows analysts to verify the findings.

Datasets

The primary dataset used for evaluation consists of 482 critical CVEs reported in 2024 with a CVSS score of 9.0 or higher. These CVEs were specifically chosen for their direct mapping to the CWE database, which provides mitigation strategy information. The system also leverages publicly available data from:

  • NVD (National Vulnerability Database)
  • CWE (Common Weakness Enumeration)
  • Aqua Vulnerability Database (Aqua) - This database was found to provide particularly structured and detailed context, significantly improving performance.
  • Various hyperlinks found on NVD entries, which provided additional relevant information.

Key Insights and Ideas

  • Summarization beats Chunking: The paper demonstrates that a summarizing retrieval technique is more effective than traditional chunking for vulnerability analysis, especially for complex queries like mitigation strategies. It handles large amounts of data more efficiently and accurately.
  • Provenance is Key to Trust: By explicitly linking LLM outputs to verifiable sources, ProveRAG builds trust. Analysts can see why a recommendation was made, not just what it is.
  • Self-Critique Enhances Accuracy: The evaluation module's ability to identify TPs, FPs, and FNs, and provide rationales, helps refine the LLM's output and identify its weaknesses.
  • Structured Data is Powerful: Integrating structured databases like Aqua significantly boosts the LLM's performance, highlighting the importance of data quality and organization.
  • LLM Choice Matters: While ProveRAG is designed to be generalizable, the performance can vary based on the LLM's instruction-following capabilities. More powerful models like GPT-4o-mini generally perform better.

Key Results

ProveRAG achieved impressive accuracy rates:

  • Exploitation: 99% True Positives (TPs) when using the Aqua database.
  • Mitigation: 97% True Positives (TPs) when using the Aqua database.
  • These results significantly outperform baseline methods like prompt-only LLMs (which showed as low as 6% accuracy for mitigation) and chunking retrieval techniques.
  • The system effectively identified hallucinations (FPs) and omissions (FNs), demonstrating its self-critique capabilities.
  • Evaluation using Rouge-L and Cosine Similarity metrics confirmed that ProveRAG's outputs align well with verifiable evidence.

Limitations and Future Work

  • LLM Instruction Following: The paper notes that LLMs with weaker instruction-following capabilities (like Llama-3.1-8B) may struggle to adhere to the structured output format required for provenance and self-critique, necessitating more careful prompt engineering.
  • Computational Budget: Testing with larger open-source LLMs was limited by computational resources.
  • Refining FP/FN Detection: The authors suggest that more adaptive prompting could help reduce misclassifications of FPs and FNs, especially when dealing with very lengthy contexts.
  • Future Work: The authors aim to further explore the generalizability of ProveRAG across a wider range of LLMs and investigate more advanced prompt engineering techniques to optimize performance for different models. They also plan to continue refining the system to improve its ability to handle complex queries and provide even more robust audit trails.

Summary updated

Abstract

In cybersecurity, security analysts constantly face the challenge of mitigating newly discovered vulnerabilities in real-time, with over 300,000 vulnerabilities identified since 1999. The sheer volume of known vulnerabilities complicates the detection of patterns for unknown threats. While LLMs can assist, they often hallucinate and lack alignment with recent threats. Over 40,000 vulnerabilities have been identified in 2024 alone, which are introduced after most popular LLMs' (e.g., GPT-5) training data cutoff. This raises a major challenge of leveraging LLMs in cybersecurity, where accuracy and up-to-date information are paramount. Therefore, we aim to improve the adaptation of LLMs in vulnerability analysis by mimicking how an analyst performs such tasks. We propose ProveRAG, an LLM-powered system designed to assist in rapidly analyzing vulnerabilities with automated retrieval augmentation of web data while self-evaluating its responses with verifiable evidence. ProveRAG incorporates a self-critique mechanism to help alleviate the omission and hallucination common in the output of LLMs applied in cybersecurity applications. The system cross-references data from verifiable sources (NVD and CWE), giving analysts confidence in the actionable insights provided. Our results indicate that ProveRAG excels in delivering verifiable evidence to the user with over 99% and 97% accuracy in exploitation and mitigation strategies, respectively. ProveRAG guides analysts to secure their systems more effectively by overcoming temporal and context-window limitations while also documenting the process for future audits.
BibTeX
@article{fayyazi2025proverag,
  author = {Reza Fayyazi and Stella Hoyos Trueba and Michael Zuzak and Shanchieh Jay Yang},
  title = {{ProveRAG: Provenance-Driven Vulnerability Analysis With Automated Retrieval-Augmented LLMs}},
  journal = {IEEE Access},
  year = {2025},
  doi = {10.1109/access.2025.3638251},
}
Figure