TL;DR
NeuroGenPoisoning is a novel white-box attack framework against Retrieval-Augmented Generation (RAG) systems that systematically overrides an LLM's internal memory. By identifying "Poison-Responsive Neurons" via gradient attribution and using a genetic algorithm to evolve adversarial passages, it achieves a >90% success rate in forcing models to output incorrect answers, even when they strongly remember the truth.
What Problem Does This Paper Address?
Retrieval-Augmented Generation (RAG) helps Large Language Models (LLMs) stay factually accurate and up-to-date by pulling in relevant external documents during inference. However, this architectural strength introduces a severe vulnerability: if an attacker injects poisoned documents into the retrieval pool, they can manipulate the LLM's final output.
Existing RAG poisoning attacks typically rely on manual prompt crafting or rigid, pre-defined templates. These methods suffer from poor scalability and frequently fail when faced with knowledge conflict—a scenario where the LLM's internal parametric memory strongly contradicts the poisoned external text, causing the model to default back to the correct internal answer. This paper addresses how to systematically exploit the internal representation dynamics of LLMs to reliably override their internal knowledge at scale.
Methods
The authors introduce NeuroGenPoisoning, an adversarial framework that combines interpretability techniques with evolutionary computing:
- Identifying Poison-Responsive Neurons: Using Integrated Gradients (IG), the method calculates attribution scores for individual neurons by comparing model activations when external context is provided versus a query-only baseline. It isolates a global set of top-$r$ neurons ($N_{top-r}$) that are consistently highly sensitive to external data across multiple queries.
- Genetic Optimization: The attack initializes a population of plausible-looking but factually incorrect passages using an LLM (GPT-4). It then runs a genetic algorithm (encompassing crossover, mutation, and selection) to iteratively evolve these passages.
- Neuron-Guided Fitness Function: Instead of optimizing for surface-level text features or semantic similarity, the fitness function directly targets the average activation score of the pre-identified Poison-Responsive Neurons. This forces the text to evolve structural and stylistic features that maximize internal model compliance.
Datasets
The framework was evaluated across three widely used open-domain question-answering (QA) benchmarks containing diverse topics (history, science, sports, and popular culture):
- SQuAD 2.0: A large-scale benchmark derived from Wikipedia articles.
- TriviaQA: A reading comprehension dataset containing question-answer-evidence triples.
- WikiQA: A dataset consisting of real Bing query logs paired with candidate answer sentences from Wikipedia.
Key Insights and Ideas
- Targeting Internals over Semantics: Traditional attacks optimize for token-level changes or semantic similarity to the query. NeuroGenPoisoning demonstrates that optimizing directly for internal neuron attribution signals is far more effective, effectively creating a pathway that forces context reliance.
- Reusing Promising Failures: In standard genetic algorithms, candidate texts that fail to alter the model's output are discarded. By analyzing neuron activation, NeuroGenPoisoning can identify "promising failures"—passages that didn't flip the output token yet but successfully activated the target neurons—allowing their traits to be preserved and bred into subsequent generations.
- Defeating Knowledge Conflict: By specifically targeting the exact internal pathways responsible for integrating external knowledge, the optimization successfully suppresses the model's strong internal memory of correct facts.
Key Results
- High Success Rate: Across multiple open-source LLMs (LLaMA-2-7b, Vicuna-7b/13b, Gemma-7b), the final Population Overwrite Success Rate (POSR) consistently exceeded 90%, up from an initial unoptimized baseline of roughly 40-50%.
- Fluency and Stealthiness: The optimization process did not degrade text quality. The perplexity (PPL) of the adversarial passages actually dropped by 2.7% to 7.7% from the initial generation, indicating that the final poisoned text became more fluent and harder to detect via automated perplexity filters.
- Massive Scalability: Unlike baseline methods like PoisonedRAG that focus on finding a single adversarial passage per query, this approach successfully scales to generate thousands of diverse, high-fitness poisoned passages per query.
Limitations and Future Work
- White-Box Assumption: The current framework requires white-box inference-time access to compute gradients and neuron activations (Integrated Gradients). This limits its immediate applicability against proprietary, closed-source APIs like OpenAI's GPT-4 or Anthropic's Claude.
- Future Directions: The authors note that extending this method to fully black-box settings is a critical next step. They plan to explore ways to estimate neuron sensitivity without gradient access, potentially by utilizing surrogate models or surface-level neuron activation proxies.
Summary updated