ControlNET: A Firewall for RAG-based LLM System

Hongwei Yao, Haoran Shi, Yidou Chen, Yixin Jiang, Cong Wang, Zhan QinarXiv 2025 · DOI · arXiv

Introduction

Retrieval-Augmented Generation (RAG) has become a cornerstone for deploying Large Language Models (LLMs) in enterprise environments, effectively reducing hallucinations by grounding responses in external knowledge bases. However, this architecture introduces significant security vulnerabilities. By allowing models to consult external corpora, RAG systems open new vectors for data breaches and poisoning attacks.

ControlNET is introduced as a novel "AI firewall" designed to secure these systems. Unlike traditional guardrails that rely on surface-level text filtering, ControlNET monitors internal LLM activation patterns to detect malicious intent and uses a plug-and-play hyper-network to steer the model toward safe, privacy-preserving responses.

The Vulnerabilities of RAG Systems

The integration of external data into the LLM inference loop creates two primary categories of risk:

  1. Data Breaching Risk: Malicious users may employ sophisticated prompting (reconnaissance) to leak system instructions or gain unauthorized access to sensitive documents within the vector database—for instance, a hospital doctor attempting to access restricted financial records.
  2. Data Poisoning Risk: Adversaries can inject "poisoned" documents into the knowledge corpus to spread misinformation or hijack the model's conversational flow.
Overview of RAG pipeline vulnerabilities including reconnaissance, data exfiltration, and knowledge poisoning.

Existing defenses, such as regex filters or standard prompt-based guardrails, often struggle with the semantic complexity of "jailbreak" attempts and the nuanced access control required in multi-user environments.

Methodology: Defense via Activation Monitoring

ControlNET shifts the defensive focus from the textual level to the activation level. The core insight is that when an LLM processes a malicious or unauthorized query, its internal neurons fire in a distinct pattern compared to benign requests, a phenomenon termed the activation shift.

1. Risk Detection via Activation Shift Index (ASI)

The system operates in two main steps for detection:

  • Anchor Activation Extracting: During a setup phase, the system records "normal" activation patterns from benign queries to establish a "whitelist" zone in the model's latent space.
  • Query Matching: For every incoming query, ControlNET calculates an Activation Shift Index (ASI). If the internal activation vector deviates significantly from the authorized zone, the query is flagged as a risk.

2. Risk Mitigation via ProNet

Rather than simply blocking flagged queries—which can disrupt user experience—ControlNET employs ProNet (Programmable Hyper-Network). This lightweight, modular sub-network adds a "correction term" to the LLM’s hidden states. This effectively "steers" the model’s behavior away from harmful outputs and toward safe, pre-defined responses (e.g., "Permissions denied") without requiring expensive retraining of the base LLM.

The ControlNET framework: Step 1 involves activation extraction and matching; Step 2 involves ProNet training and query controlling.

Experimental Results

The researchers evaluated ControlNET using standard benchmarks (MS MARCO, HotpotQA, FinQA) and a newly developed MedicalSys dataset, which simulates a multi-role hospital system with 20,000 samples to test unauthorized access across different administrative roles.

  • Detection Performance: ControlNET achieved an average AUROC of 0.974 across all tested risks. In specific categories like data exfiltration and conversation hijacking, the AUROC exceeded 0.99, outperforming existing prompt injection defenses by over 12%.
  • Utility Preservation: A critical requirement for any firewall is maintaining the model's original utility. Using the BERTScore metric, the authors demonstrated that the model's performance on legitimate tasks remained virtually unchanged, with a negligible F1-score drop of only 0.02.
  • Layer Sensitivity: The study found that lower-level layers (e.g., Layer 1 or 8) are most sensitive to risk-related perturbations, as they preserve more of the raw input signal before it is abstracted in deeper layers.

Conclusion and Future Directions

ControlNET provides a robust, activation-based defense mechanism for RAG systems that is harder to bypass than traditional text-based filters. While highly effective for standard query-response flows, future work will focus on extending this framework to complex agentic workflows involving multi-step loops and enhancing the granularity of access control within retrieved documents. By leveraging the internal "semantic divergence" of the model, ControlNET offers a promising path toward truly secure enterprise AI.

Summary updated

Abstract

Retrieval-Augmented Generation (RAG) has significantly enhanced the factual accuracy and domain adaptability of Large Language Models (LLMs). This advancement has enabled their widespread deployment across sensitive domains such as healthcare, finance, and enterprise applications. RAG mitigates hallucinations by integrating external knowledge, yet introduces privacy risk and security risk, notably data breaching risk and data poisoning risk. While recent studies have explored prompt injection and poisoning attacks, there remains a significant gap in comprehensive research on controlling inbound and outbound query flows to mitigate these threats. In this paper, we propose an AI firewall, ControlNET, designed to safeguard RAG-based LLM systems from these vulnerabilities. ControlNET controls query flows by leveraging activation shift phenomena to detect adversarial queries and mitigate their impact through semantic divergence. We conduct comprehensive experiments on four different benchmark datasets including Msmarco, HotpotQA, FinQA, and MedicalSys using state-of-the-art open source LLMs (Llama3, Vicuna, and Mistral). Our results demonstrate that ControlNET achieves over 0.909 AUROC in detecting and mitigating security threats while preserving system harmlessness. Overall, ControlNET offers an effective, robust, harmless defense mechanism, marking a significant advancement toward the secure deployment of RAG-based LLM systems.
BibTeX
@misc{yao2025controlnet,
  author = {Hongwei Yao and Haoran Shi and Yidou Chen and Yixin Jiang and Cong Wang and Zhan Qin},
  title = {{ControlNET: A Firewall for RAG-based LLM System}},
  year = {2025},
  eprint = {2504.09593},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.09593},
}
Figure