NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledge

Hanyu Zhu, Lance Fiondella, Jiawei Yuan, Kai Zeng, Long Jiao

TL;DR

NeuroGenPoisoning is a novel white-box attack framework against Retrieval-Augmented Generation (RAG) systems that systematically overrides an LLM's internal memory. By identifying "Poison-Responsive Neurons" via gradient attribution and using a genetic algorithm to evolve adversarial passages, it achieves a >90% success rate in forcing models to output incorrect answers, even when they strongly remember the truth.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) helps Large Language Models (LLMs) stay factually accurate and up-to-date by pulling in relevant external documents during inference. However, this architectural strength introduces a severe vulnerability: if an attacker injects poisoned documents into the retrieval pool, they can manipulate the LLM's final output.

Existing RAG poisoning attacks typically rely on manual prompt crafting or rigid, pre-defined templates. These methods suffer from poor scalability and frequently fail when faced with knowledge conflict—a scenario where the LLM's internal parametric memory strongly contradicts the poisoned external text, causing the model to default back to the correct internal answer. This paper addresses how to systematically exploit the internal representation dynamics of LLMs to reliably override their internal knowledge at scale.

Figure 2: The workflow of attack against RAG systems based on Poison-Responsive Neurons.

Methods

The authors introduce NeuroGenPoisoning, an adversarial framework that combines interpretability techniques with evolutionary computing:

  1. Identifying Poison-Responsive Neurons: Using Integrated Gradients (IG), the method calculates attribution scores for individual neurons by comparing model activations when external context is provided versus a query-only baseline. It isolates a global set of top-$r$ neurons ($N_{top-r}$) that are consistently highly sensitive to external data across multiple queries.
  2. Genetic Optimization: The attack initializes a population of plausible-looking but factually incorrect passages using an LLM (GPT-4). It then runs a genetic algorithm (encompassing crossover, mutation, and selection) to iteratively evolve these passages.
  3. Neuron-Guided Fitness Function: Instead of optimizing for surface-level text features or semantic similarity, the fitness function directly targets the average activation score of the pre-identified Poison-Responsive Neurons. This forces the text to evolve structural and stylistic features that maximize internal model compliance.

Datasets

The framework was evaluated across three widely used open-domain question-answering (QA) benchmarks containing diverse topics (history, science, sports, and popular culture):

  • SQuAD 2.0: A large-scale benchmark derived from Wikipedia articles.
  • TriviaQA: A reading comprehension dataset containing question-answer-evidence triples.
  • WikiQA: A dataset consisting of real Bing query logs paired with candidate answer sentences from Wikipedia.

Key Insights and Ideas

  • Targeting Internals over Semantics: Traditional attacks optimize for token-level changes or semantic similarity to the query. NeuroGenPoisoning demonstrates that optimizing directly for internal neuron attribution signals is far more effective, effectively creating a pathway that forces context reliance.
  • Reusing Promising Failures: In standard genetic algorithms, candidate texts that fail to alter the model's output are discarded. By analyzing neuron activation, NeuroGenPoisoning can identify "promising failures"—passages that didn't flip the output token yet but successfully activated the target neurons—allowing their traits to be preserved and bred into subsequent generations.
  • Defeating Knowledge Conflict: By specifically targeting the exact internal pathways responsible for integrating external knowledge, the optimization successfully suppresses the model's strong internal memory of correct facts.

Key Results

  • High Success Rate: Across multiple open-source LLMs (LLaMA-2-7b, Vicuna-7b/13b, Gemma-7b), the final Population Overwrite Success Rate (POSR) consistently exceeded 90%, up from an initial unoptimized baseline of roughly 40-50%.
  • Fluency and Stealthiness: The optimization process did not degrade text quality. The perplexity (PPL) of the adversarial passages actually dropped by 2.7% to 7.7% from the initial generation, indicating that the final poisoned text became more fluent and harder to detect via automated perplexity filters.
  • Massive Scalability: Unlike baseline methods like PoisonedRAG that focus on finding a single adversarial passage per query, this approach successfully scales to generate thousands of diverse, high-fitness poisoned passages per query.
Figure 3: POSR Across Models and Datasets of NeuroGenPoisoning

Limitations and Future Work

  • White-Box Assumption: The current framework requires white-box inference-time access to compute gradients and neuron activations (Integrated Gradients). This limits its immediate applicability against proprietary, closed-source APIs like OpenAI's GPT-4 or Anthropic's Claude.
  • Future Directions: The authors note that extending this method to fully black-box settings is a critical next step. They plan to explore ways to estimate neuron sensitivity without gradient access, potentially by utilizing surrogate models or surface-level neuron activation proxies.

Summary updated

Abstract

Retrieval-Augmented Generation (RAG) empowers Large Language Models (LLMs) to dynamically integrate external knowledge during inference, improving their factual accuracy and adaptability. However, adversaries can inject poisoned external knowledge to override the model's internal memory. While existing attacks iteratively manipulate retrieval content or prompt structure of RAG, they largely ignore the model's internal representation dynamics and neuron-level sensitivities. The underlying mechanism of RAG poisoning has not been fully studied and the effect of knowledge conflict with strong parametric knowledge in RAG is not considered. In this work, we propose NeuroGenPoisoning, a novel attack framework that generates adversarial external knowledge in RAG guided by LLM internal neuron attribution and genetic optimization. Our method first identifies a set of Poison-Responsive Neurons whose activation strongly correlates with contextual poisoning knowledge. We then employ a genetic algorithm to evolve adversarial passages that maximally activate these neurons. Crucially, our framework enables massive-scale generation of effective poisoned RAG knowledge by identifying and reusing promising but initially unsuccessful external knowledge variants via observed attribution signals. At the same time, Poison-Responsive Neurons guided poisoning can effectively resolves knowledge conflict. Experimental results across models and datasets demonstrate consistently achieving high Population Overwrite Success Rate (POSR) of over 90% while preserving fluency. Empirical evidence shows that our method effectively resolves knowledge conflict.
BibTeX
@inproceedings{zhu2025neurogenpoisoning,
  author = {Hanyu Zhu and Lance Fiondella and Jiawei Yuan and Kai Zeng and Long Jiao},
  title = {{NeuroGenPoisoning: Neuron-Guided Attacks on Retrieval-Augmented Generation of LLM via Genetic Optimization of External Knowledge}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=4kTpb8pITI},
}
Figure