Abstract
This research identifies a critical vulnerability in Large Language Models (LLMs) stemming from their continuous activation space. By perturbing the internal hidden states of just a few tokens, an attacker can exert precise control over long sequences of generated text. We establish a linear scaling law for these "activation attacks," demonstrating that the number of controllable tokens scales predictably with the number of manipulated tokens and the model's activation dimension. This vulnerability is fundamentally linked to a dimensionality mismatch between high-dimensional internal representations and the discrete token output space.
Introduction
Adversarial research in machine learning has traditionally focused on discrete input modifications, such as pixel-level changes in images or word substitutions in text. However, the rise of multi-modal models and retrieval-augmented generation (RAG) has expanded the attack surface to include direct manipulation of internal model activations. This paper quantifies the potency of these activation attacks and explores whether their effectiveness follows predictable scaling laws as models increase in size and complexity.
Methodology
The study introduces the concept of activation attacks, where a small perturbation vector ($P$) is added to the continuous hidden states of the initial tokens in a sequence.
The perturbation $P$ is optimized using the Adam algorithm to minimize the cross-entropy loss between the model's output and a target sequence of tokens. To ensure the findings are robust, the researchers evaluated several model families, including EleutherAI’s Pythia (ranging from 70M to 2.8B parameters) and Microsoft’s Phi models. Target sequences were generated using randomly sampled tokens to avoid linguistic bias.
Theoretical Framework: Dimensionality Mismatch
The core hypothesis is that LLMs are inherently vulnerable due to a dimensionality mismatch. The internal activation space ($d$) has far more degrees of freedom than the discrete output space of tokens. The paper defines attack resistance ($\chi$) as the ratio of input control bits required to flip a single bit of output.
Empirical results show that $\chi$ remains remarkably constant (between 16 and 25) across two orders of magnitude of model size. This implies that as models grow and their internal dimensions ($d$) increase, they do not become more robust; rather, they become susceptible to even longer-range control because the "budget" for manipulation increases with $d$.
Empirical Results and Scaling Laws
The primary finding of this work is the confirmation of a linear scaling law: $$t_{max} = \kappa a$$ where $t_{max}$ is the maximum number of tokens an attacker can control, $a$ is the number of tokens whose activations are manipulated, and $\kappa$ is the "attack multiplier."
Key findings include:
- Potency: Activation attacks are significantly more powerful than token-level attacks. For a 70M parameter model, the attack multiplier $\kappa$ is ~24.2, compared to just ~0.12 for greedy token substitution—a 200-fold increase in effectiveness.
- Predictability: The multiplier $\kappa$ scales linearly with the model's activation dimension $d$, allowing for the prediction of vulnerability in future, larger models.
- Persistence: The influence of a perturbation does not decay rapidly; an attack at the beginning of a prompt can precisely dictate tokens generated over 100 positions later.
Discussion and Limitations
While this study establishes a clear theoretical and empirical baseline for activation attacks, several areas remain for future exploration. The use of random token targets suggests a worst-case scenario; further research is needed to determine if semantic constraints in natural language targets alter these scaling laws. Additionally, while the attacks were successful across various layers, a full "depth-profile" of vulnerability has yet to be mapped. Finally, the research highlights the need for novel defense mechanisms, such as activation quantization, to mitigate the risks posed by the high-dimensional nature of LLM internal states.
Summary updated