NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledge

Hanyu Zhu, Lance Fiondella, Jiawei Yuan, Kai Zeng, Long Jiao

Abstract

NeuroGenPoisoning is a sophisticated attack framework designed to compromise Retrieval-Augmented Generation (RAG) systems by overriding a Large Language Model's (LLM) internal parametric memory. By identifying "Poison-Responsive Neurons" (PRNs) sensitive to external context, the framework employs genetic algorithms to evolve adversarial passages. Evaluation across SQuAD 2.0, TriviaQA, and WikiQA benchmarks demonstrates a Population Overwrite Success Rate (POSR) exceeding 90% against models like LLaMA-2 and Vicuna, while preserving text fluency and stealth.

Introduction

Retrieval-Augmented Generation (RAG) has become the industry standard for grounding Large Language Models (LLMs) in up-to-date, external information. However, this reliance introduces a critical vulnerability: knowledge poisoning. When external data contradicts a model's internal training data, a "knowledge conflict" occurs. Traditional attacks often fail because they rely on simple templates and do not account for the internal mechanisms LLMs use to resolve these conflicts. NeuroGenPoisoning addresses this by targeting the model's internal representation dynamics, ensuring that poisoned external knowledge consistently overrides internal "beliefs."

Methodology: Exploiting Internal Gatekeepers

The NeuroGenPoisoning framework operates through a multi-stage process that combines interpretability techniques with evolutionary computation.

The NeuroGenPoisoning attack framework targets specific internal neurons to resolve knowledge conflicts in favor of poisoned external data.
  1. Identifying Poison-Responsive Neurons (PRNs): Using a technique called Integrated Gradients (IG), the authors identify a small subset of neurons (often just the top 10) that act as "gatekeepers," determining when the model should trust external context over its own memory.
  2. Adversarial Seed Generation: Initial "seed" passages containing target misinformation are generated using high-capability models like GPT-4 to ensure they appear as legitimate Wikipedia articles or news reports.
  3. Genetic Optimization: These seeds are evolved using a genetic algorithm where "fitness" is defined by the activation strength of the identified PRNs rather than surface-level text features.
  4. Neuron-Guided Evolution: The process uses internal neuron attribution signals to refine the population, ensuring that even unsuccessful passages contributing to the right internal circuits are preserved for future generations.

Experimental Evaluation

The framework was evaluated across three major open-domain Question Answering (QA) benchmarks: SQuAD 2.0, TriviaQA, and WikiQA. The target models included popular open-source architectures such as LLaMA-2-7b, Vicuna-7b/13b, and Gemma-7b.

Key Results

The optimization process proved highly effective, nearly doubling the success rate of initial seed passages within just 10 generations.

The Population Overwrite Success Rate (POSR) consistently reaches over 90% across various models and datasets within 10 generations of optimization.
  • High Success Rate: The method achieved a POSR of over 90% across all tested models and datasets.
  • Stealth and Fluency: Evolved passages maintained high fluency, as measured by perplexity metrics, making them difficult to detect via standard automated filters or human review.
  • Overcoming Strong Internal Memory: The attack successfully forced models to output incorrect answers even when the model had a strong internal preference for the correct fact (e.g., changing the model's "belief" about historical figures).

Conclusion and Future Work

NeuroGenPoisoning exposes a significant vulnerability in RAG systems by demonstrating how internal model interpretability can be weaponized to bypass safety and accuracy guardrails. While currently a white-box attack requiring access to model gradients, it highlights the urgent need for "neuron-aware" defensive strategies. Future research will focus on black-box transferability—using open models to attack proprietary ones—and the development of robust filters capable of detecting these optimized adversarial passages.

Summary updated

Abstract

Retrieval-Augmented Generation (RAG) empowers Large Language Models (LLMs) to dynamically integrate external knowledge during inference, improving their factual accuracy and adaptability. However, adversaries can inject poisoned external knowledge to override the model’s internal memory. While existing attacks iteratively manipulate retrieval content or prompt structure of RAG, they largely ignore the model’s internal representation dynamics and neuron-level sensitivities. The underlying mechanism of RAG poisoning has not been fully studied and the effect of knowledge conflict with strong parametric knowledge in RAG is not considered. In this work, we propose NeuroGenPoisoning, a novel attack framework that generates adversarial external knowledge in RAG guided by LLM internal neuron attribution and genetic optimization. Our method first identifies a set of **Poison-Responsive Neurons** whose activation strongly correlates with contextual poisoning knowledge. We then employ a genetic algorithm to evolve adversarial passages that maximally activate these neurons. Crucially, our framework enables massive-scale generation of effective poisoned RAG knowledge by identifying and reusing promising but initially unsuccessful external knowledge variants via observed attribution signals. At the same time, Poison-Responsive Neurons guided poisoning can effectively resolves knowledge conflict. Experimental results across models and datasets demonstrate consistently achieving high Population Overwrite Success Rate (POSR) of over 90\% while preserving fluency. Empirical evidence shows that our method effectively resolves knowledge conflict.
BibTeX
@inproceedings{zhu2025neurogenpoisoning,
  author = {Hanyu Zhu and Lance Fiondella and Jiawei Yuan and Kai Zeng and Long Jiao},
  title = {{NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledge}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=4kTpb8pITI},
}
Figure