Abstract
As large language models (LLMs) become integrated into various sensitive applications, prompt injection, the use of prompting to induce harmful behaviors from LLMs, poses an ever increasing risk. Prompt injection attacks can cause LLMs to leak sensitive data, spread misinformation, and exhibit harmful behaviors. To defend against these attacks, we propose CourtGuard, a locally-runnable, multiagent prompt injection classifier. In it, prompts are evaluated in a court-like multiagent LLM system, where a "defense attorney" model argues the prompt is benign, a "prosecution attorney" model argues the prompt is a prompt injection, and a "judge" model gives the final classification. CourtGuard has a lower false positive rate than the Direct Detector, an LLM as-a-judge. However, CourtGuard is generally a worse prompt injection detector. Nevertheless, this lower false positive rate highlights the importance of considering both adversarial and benign scenarios for the classification of a prompt. Additionally, the relative performance of CourtGuard in comparison to other prompt injection classifiers advances the use of multiagent systems as a defense against prompt injection attacks.
Introduction
As Large Language Models (LLMs) are integrated into systems handling sensitive information—such as medical records, financial data, and military intelligence—they become prime targets for prompt injection attacks. These attacks can hijack the model to leak private data, spread misinformation, or execute unauthorized actions. While enterprise defenses exist, they often require sending potentially sensitive user data to external APIs, which is a significant privacy concern for many organizations. There is a pressing need for simple, effective, and locally runnable defenses that developers can deploy internally to keep sensitive data secure.
Methodology
We propose CourtGuard, a multiagent LLM system designed to run locally and in parallel with a main LLM application. CourtGuard utilizes a courtroom-inspired architecture to classify incoming prompts:
- Defense Attorney Model: Argues why the input prompt is benign and safe.
- Prosecution Attorney Model: Argues why the input is a malicious prompt injection.
- Judge Model: Reviews both arguments and issues a final classification.
This multiagent setup is compared against a baseline Direct Detector, which is a single LLM asked to act as a judge and classify the prompt directly. To ensure the system can run locally, we utilized smaller, open-weight models: Gemma-3-12b-it, Llama-3.3-8B, and Phi-4-mini-instruct.
Experimental Setup
The system was evaluated on three datasets to test both attack detection (true positives) and over-defensiveness (false positives):
- LLMail-Inject: A sample of 5,000 real-world, indirect prompt injection attacks sourced from an email-based challenge.
- NotInject: A dataset of benign prompts containing "trigger words" typically associated with prompt injections, designed to test for false positives.
- Qualifire Prompt Injection Benchmark: A mixed dataset containing both benign prompts and adversarial jailbreak attempts.
Results and Discussion
Our evaluation revealed a significant trade-off between sensitivity and specificity when comparing the multiagent approach to direct detection.
Benign Prompt Accuracy
CourtGuard generally outperformed the Direct Detector on benign prompts. By forcing the "judge" model to read a defense of the prompt, the system is much less likely to flag a benign prompt as an attack. For example, on the NotInject dataset, Llama-3.3-8B scored 94.55% with CourtGuard versus 90.61% with the Direct Detector.
Attack Detection and Overall Performance
While CourtGuard excels at reducing false positives, the Direct Detector was vastly superior at catching actual attacks. On the LLMail-Inject dataset, the Direct Detector (using Llama) caught 59.82% of attacks, while CourtGuard caught only 14.88%. Consequently, the Direct Detector achieved higher F1 scores across most benchmarks, indicating it remains a more robust general-purpose classifier.
Insights
Qualitative analysis suggests that standard direct detectors may rely on "hidden reasoning"—non-verbalized encodings from training data—that allows them to spot attacks intuitively but makes them prone to over-defensiveness. CourtGuard's multiagent system trades this intuitive confidence for nuance, explicitly weighing both sides to reduce false positives at the cost of overall recall.
Conclusion
CourtGuard demonstrates the potential of multiagent systems to provide more balanced prompt classification, particularly in reducing false positives for benign inputs. However, the current implementation faces challenges regarding high latency and lower recall on actual attacks. Future research should aim to combine the strengths of both approaches: leveraging the intuitive detection of direct models while maintaining the deliberative safety of multiagent systems.
The implementations of CourtGuard and the Direct Detector are available at https://github.com/isaacwu2000/CourtGuard.
Summary updated