Adversarial Evasion on LLMs

Rachid Guerraoui, Rafaël PinotOpenalex Unknown 2024 · DOI · OpenAlex

Abstract

While Machine Learning (ML) applications have shown impressive achievements in tasks such as computer vision, NLP, and control problems, such achievements were possible, first and foremost, in the best-case-scenario setting. Unfortunately, settings where ML applications fail unexpectedly, abound, and malicious ML application users or data contributors can trigger such failures. This problem became known as adversarial example robustness. While this field is in rapid development, some fundamental results have been uncovered, allowing some insight into how to make ML methods resilient to input and data poisoning. Such ML applications are termed adversarially robust. While the current generation of LLMs is not adversarially robust, results obtained in other branches of ML can provide insight into how to make them adversarially robust. Such insight would complement and augment ongoing empirical efforts in the same direction (red-teaming).

Introduction

Machine Learning (ML) models are notoriously fragile. While they perform impressively in "best-case" scenarios, they often fail catastrophically when faced with adversarial examples—inputs specifically designed by a malicious actor to fool the model. As Large Language Models (LLMs) become omnipresent in industry and public sectors, their vulnerability to evasion attacks (attacks at inference time) raises serious questions about legal liability and safety.

The classic 'panda-to-gibbon' adversarial attack demonstrates how imperceptible noise can cause catastrophic misclassification.

This research synthesizes the evolution of adversarial evasion attacks, tracing their trajectory from simple image misclassifications to the complex "jailbreaking" of LLMs. It argues that while current LLM defenses rely heavily on empirical red-teaming, true resilience requires a shift toward the theoretical frameworks and provable robustness developed in other branches of machine learning.

Methodologies: Attack and Defense

The field of adversarial robustness categorizes strategies into two main camps: attack generation and defensive hardening.

Attack Strategies

Attackers often employ gradient-based methods, leveraging the model's own gradients to identify the "path of least resistance" toward an erroneous output. By applying optimization schemes, they can create imperceptible perturbations that shift an input across a decision boundary.

Defense Frameworks

  • Adversarial Training: An empirical "brute force" approach where models are explicitly trained on adversarial examples to improve recognition.
  • Provable Robustness: A formal approach using mathematical analysis—such as randomized smoothing or problem relaxation—to provide theoretical guarantees. This ensures a model's prediction remains constant within a defined "safety bubble" around an input.

In the context of LLMs, evasion often takes the form of "jailbreaking," where prompts are engineered to bypass safety filters. While the discrete nature of text makes gradient-based optimization difficult, multi-modal models (combining text and images) remain highly vulnerable to image-based perturbations.

Key Insights and Theoretical Frameworks

Several fundamental principles govern the difficulty of achieving robustness:

  • The Curse of Dimensionality: As the number of trainable parameters increases, robustness becomes harder to achieve. This is a significant concern for LLMs with billions of parameters.
  • Game-Theoretic Framing: Robustness is best understood as a two-player zero-sum game (a minimax problem). The model seeks to minimize error, while the attacker seeks to maximize it. This perspective suggests that standard learning theory is insufficient for solving adversarial vulnerabilities.
  • Computational Complexity: Creating robust models is not merely a data problem; it is a matter of computational hardness. Finding a truly robust classifier is often significantly more complex than the original learning task.
  • The Multi-modal Backdoor: Currently, the most effective "jailbreaks" may not be clever text prompts but rather adversarial images in multi-modal inputs, exploiting well-documented vulnerabilities in computer vision.

Results and Performance Gaps

Empirical results highlight a stark "robustness tax":

  • High Failure Rates: Undefended models typically have a 99% chance of failure when presented with a targeted adversarial example.
  • The Performance Gap: Even the most advanced "provable" defenses on standard benchmarks like ImageNet only achieve approximately 70% accuracy under attack.
  • Textual Resilience: Pure text-based evasion is currently less effective than image-based attacks due to the discrete nature of language, though manual jailbreaks remain a practical threat.

Limitations and Future Directions

Current theoretical results are often pessimistic, suggesting that our mathematical definitions of adversarial attacks might be ill-posed by assuming an unrealistically powerful attacker.

Future research should focus on refining frameworks to develop more realistic mathematical definitions of real-world adversarial examples and creating new theoretical models to handle the discrete, high-dimensional nature of language. Ultimately, bridging the gap between the well-understood world of adversarial images and the emerging frontier of LLMs is essential for building trustworthy AI systems.

Summary updated

Abstract

Abstract While Machine Learning (ML) applications have shown impressive achievements in tasks such as computer vision, NLP, and control problems, such achievements were possible, first and foremost, in the best-case-scenario setting. Unfortunately, settings where ML applications fail unexpectedly, abound, and malicious ML application users or data contributors can trigger such failures. This problem became known as adversarial example robustness. While this field is in rapid development, some fundamental results have been uncovered, allowing some insight into how to make ML methods resilient to input and data poisoning. Such ML applications are termed adversarially robust . While the current generation of LLMs is not adversarially robust, results obtained in other branches of ML can provide insight into how to make them adversarially robust. Such insight would complement and augment ongoing empirical efforts in the same direction (red-teaming).
BibTeX
@inproceedings{guerraoui2024adversarial,
  author = {Rachid Guerraoui and Rafaël Pinot},
  title = {{Adversarial Evasion on LLMs}},
  booktitle = {Openalex Unknown},
  year = {2024},
  doi = {10.1007/978-3-031-54827-7\_20},
}

Loading related content...

Figure