Abstract
Large Language Models (LLMs) are rapidly evolving from simple chatbots into "operating systems" that control application logic and data. However, they lack a fundamental cybersecurity requirement: the separation of data and instructions. This paper argues that without this separation, AI-integrated systems remain inherently vulnerable to indirect prompt injection and systemic disinformation. We propose a shift from empirical testing toward foundational safety guarantees and automated red teaming to address these systemic risks.
Introduction
The rapid integration of General-Purpose AI (GPAI) into software ecosystems—exemplified by Microsoft Copilot and OpenAI’s Altera—has created a significant security gap. While these models are granted increasing autonomy and tool-use capabilities, they are fundamentally susceptible to indirect prompt injection.
In traditional computing, "code" and "data" are strictly separated to prevent malicious data from being executed as code. In contrast, LLMs process all inputs as a unified sequence of tokens. This architectural flaw allows an attacker to embed "commands" within data (e.g., an email or a website) that the AI is tasked to process. When the AI interprets these hidden commands as legitimate instructions, it can lead to data exfiltration, unauthorized API calls, or the propagation of misinformation.
Methodology for Robust AI Safety
To address these vulnerabilities, this research synthesizes several contributions toward a more robust safety framework:
- Formalization of Data-Instruction Separation: Identifying the lack of a boundary between user instructions and external data as the root cause of injection vulnerabilities.
- Automated Red Teaming: Moving beyond static benchmarks—which are prone to overfitting and evasion—toward dynamic mechanisms that use cybersecurity "playbooks" to adaptively test model defenses.
- Foundational Guarantees: Advocating for mathematical frameworks such as Differential Privacy and Robustness Certification. Unlike empirical testing, which only addresses known attacks, these frameworks provide rigorous safety bounds by design.
Evaluation Frameworks and Benchmarks
The research utilizes several specialized benchmarks to quantify these risks:
- SEP Benchmark: Quantifies an LLM's ability to adhere to the "Data-Instruction-Separation" demand.
- ELSA Benchmarking Platform: A comprehensive repository for auditing AI models regarding robustness, privacy, and human oversight.
- LLM-Capture-the-Flag: A security-focused challenge used to evaluate model vulnerabilities in a competitive environment.
- Codelmsec Benchmark: A dataset designed to systematically identify security vulnerabilities in code generated by AI agents.
Key Insights: The AI as an Information Bottleneck
A critical shift is occurring in how information is consumed: users are increasingly relying on AI-mediated summaries (e.g., Gemini, Bing Copilot) rather than raw web content. This positions AI assistants as central "information bottlenecks," where a single injection attack can facilitate large-scale disinformation.
Furthermore, the research highlights the "Root Password" Fallacy: relying on secret, static validation sets is as precarious as a single password; once leaked, the entire safety measure fails. Additionally, the complexity of multi-agent systems increases the attack surface, as even interacting agents remain vulnerable to adversarial behavior.
Results and Threat Taxonomy
Current state-of-the-art LLMs struggle significantly with the SEP benchmark, demonstrating a persistent inability to distinguish between data to be processed and instructions to be followed. Initial results from the Codelmsec benchmark also reveal that AI code generation agents frequently introduce security vulnerabilities into the software they produce.
The research maps indirect prompt injection to a wide range of classic cybersecurity threats, including phishing, malware spreading, and denial-of-service (DoS) attacks.
Conclusion and Future Work
Defining safety in specialized fields like "AI for Science" remains challenging, as the danger is often context-dependent. Future work must address "test contamination" by creating frameworks that generate evaluation tests dynamically. Ultimately, the goal is to align AI capabilities with strategic safety agendas, such as the ELSA Strategic Research Agenda, to ensure AI development remains secure and aligned with human values.
Summary updated