Introduction
Adversarial attacks—small, intentional perturbations that cause a machine learning model to malfunction—have been studied extensively in computer vision. However, their behavior in Large Language Models (LLMs) is more complex due to the discrete nature of text. This paper investigates a fundamental question: How much control does an attacker actually have over an LLM's output?
Specifically, the research explores the "dimensionality mismatch" hypothesis, which suggests that models are vulnerable because their input space (high-dimensional continuous vectors) is vastly larger than their output space (discrete predicted tokens). By treating LLMs as a controllable test-bed, the authors quantify this vulnerability and derive a linear scaling law: the number of output tokens an attacker can precisely dictate grows linearly with the number of input activations they control.
Attack Methodologies
The researchers compared two primary attack vectors to understand the limits of adversarial control:
- Activation Attacks: The attacker injects a continuous perturbation vector directly into the model's internal activations (the "residual stream") immediately following the embedding layer. Because these activations are continuous floating-point numbers, the attacker can use gradient-based optimization (like Adam) to find the exact "nudge" needed to force a specific output sequence.
- Token Substitution Attacks: A more traditional, discrete approach where the attacker replaces input tokens with others. The authors used a greedy, exhaustive search over the vocabulary to find the optimal tokens to achieve the target output.
To probe these limits further, the study introduced "multi-attacks" (forcing one perturbation to solve multiple tasks simultaneously) and "fractional dimension attacks" (restricting the attacker to only change a subset of the vector's dimensions).
Experimental Setup
The study utilized several well-known open-source model families to ensure the findings generalized across architectures:
- EleutherAI/Pythia: Ranging from 70M to 2.8B parameters.
- Microsoft/Phi-1 and Phi-1.5: High-performing small language models (1.3B parameters).
- TinyStories-33M: A small model trained on simple synthetic stories.
Attacks were tested using both randomly sampled tokens and natural language sequences, such as forcing a math problem prompt to output a specific poem.
Scaling Laws and Attack Resistance
The core contribution of this paper is the empirical verification of scaling laws governing these attacks.
- The Linear Scaling Law: The maximum number of target tokens ($t_{max}$) an attacker can dictate follows a simple linear relationship: $t_{max} = \kappa a$, where $a$ is the number of tokens whose activations the attacker controls, and $\kappa$ is the "attack multiplier."
- Attack Resistance ($\chi$): The authors define attack resistance as the number of bits of input control needed to exert one bit of control over the output. Surprisingly, this value is remarkably stable (between $\approx 16$ and $25$) across different model architectures and sizes.
- Activations vs. Tokens: Activation attacks are predictably much stronger than token attacks. For instance, in a Pythia-70M model, controlling the activations of a single token allows an attacker to dictate ~24 output tokens. Conversely, it takes ~8 input tokens to dictate just one output token via discrete substitution.
- Dimensionality Mismatch: Despite the raw power difference, the efficiency of the control (bits of input vs. bits of output) is surprisingly similar between activation and token attacks. This strongly supports the theory that adversarial vulnerability is a direct consequence of the high dimensionality of the model's internal representations.
Key Results and Implications
The level of control demonstrated is massive. By manipulating all dimensions of a single token's activation, the authors successfully forced a model to recite a specific 40-token poem as the answer to "1 + 2 = ". Furthermore, these attacks exhibit "distance robustness," remaining effective even with a gap of up to 100 tokens between the manipulated input and the target output.
While activation attacks require "white-box" access (knowledge of internal weights and the ability to inject vectors), this poses a severe, practical risk for multi-modal models (where continuous image embeddings are converted to activations) and Retrieval-Augmented Generation (RAG) systems. In these setups, attackers might control the continuous vectors injected into the prompt, bypassing the discrete token bottleneck entirely. Future work will need to investigate how defense mechanisms, quantization, or safety training might alter a model's inherent attack resistance.
Summary updated