Scaling Laws for Adversarial Attacks on Language Model Activations

Stanislav FortarXiv 2023 · DOI · arXiv

Abstract

This research identifies a critical vulnerability in Large Language Models (LLMs) stemming from their continuous activation space. By perturbing the internal hidden states of just a few tokens, an attacker can exert precise control over long sequences of generated text. We establish a linear scaling law for these "activation attacks," demonstrating that the number of controllable tokens scales predictably with the number of manipulated tokens and the model's activation dimension. This vulnerability is fundamentally linked to a dimensionality mismatch between high-dimensional internal representations and the discrete token output space.

Introduction

Adversarial research in machine learning has traditionally focused on discrete input modifications, such as pixel-level changes in images or word substitutions in text. However, the rise of multi-modal models and retrieval-augmented generation (RAG) has expanded the attack surface to include direct manipulation of internal model activations. This paper quantifies the potency of these activation attacks and explores whether their effectiveness follows predictable scaling laws as models increase in size and complexity.

Methodology

The study introduces the concept of activation attacks, where a small perturbation vector ($P$) is added to the continuous hidden states of the initial tokens in a sequence.

Diagram of an activation attack, illustrating how small perturbations in the embedding space can shift next-token probability distributions.

The perturbation $P$ is optimized using the Adam algorithm to minimize the cross-entropy loss between the model's output and a target sequence of tokens. To ensure the findings are robust, the researchers evaluated several model families, including EleutherAI’s Pythia (ranging from 70M to 2.8B parameters) and Microsoft’s Phi models. Target sequences were generated using randomly sampled tokens to avoid linguistic bias.

Theoretical Framework: Dimensionality Mismatch

The core hypothesis is that LLMs are inherently vulnerable due to a dimensionality mismatch. The internal activation space ($d$) has far more degrees of freedom than the discrete output space of tokens. The paper defines attack resistance ($\chi$) as the ratio of input control bits required to flip a single bit of output.

Empirical results show that $\chi$ remains remarkably constant (between 16 and 25) across two orders of magnitude of model size. This implies that as models grow and their internal dimensions ($d$) increase, they do not become more robust; rather, they become susceptible to even longer-range control because the "budget" for manipulation increases with $d$.

Empirical Results and Scaling Laws

The primary finding of this work is the confirmation of a linear scaling law: $$t_{max} = \kappa a$$ where $t_{max}$ is the maximum number of tokens an attacker can control, $a$ is the number of tokens whose activations are manipulated, and $\kappa$ is the "attack multiplier."

Experimental data vs. theoretical prediction for the scaling of target tokens ($t_{max}$) relative to the attack length ($a$) and simultaneous attacks ($n$).

Key findings include:

  • Potency: Activation attacks are significantly more powerful than token-level attacks. For a 70M parameter model, the attack multiplier $\kappa$ is ~24.2, compared to just ~0.12 for greedy token substitution—a 200-fold increase in effectiveness.
  • Predictability: The multiplier $\kappa$ scales linearly with the model's activation dimension $d$, allowing for the prediction of vulnerability in future, larger models.
  • Persistence: The influence of a perturbation does not decay rapidly; an attack at the beginning of a prompt can precisely dictate tokens generated over 100 positions later.

Discussion and Limitations

While this study establishes a clear theoretical and empirical baseline for activation attacks, several areas remain for future exploration. The use of random token targets suggests a worst-case scenario; further research is needed to determine if semantic constraints in natural language targets alter these scaling laws. Additionally, while the attacks were successful across various layers, a full "depth-profile" of vulnerability has yet to be mapped. Finally, the research highlights the need for novel defense mechanisms, such as activation quantization, to mitigate the risks posed by the high-dimensional nature of LLM internal states.

Summary updated

Abstract

We explore a class of adversarial attacks targeting the activations of language models. By manipulating a relatively small subset of model activations, $a$, we demonstrate the ability to control the exact prediction of a significant number (in some cases up to 1000) of subsequent tokens $t$. We empirically verify a scaling law where the maximum number of target tokens $t_\mathrm{max}$ predicted depends linearly on the number of tokens $a$ whose activations the attacker controls as $t_\mathrm{max} = κa$. We find that the number of bits of control in the input space needed to control a single bit in the output space (what we call attack resistance $χ$) is remarkably constant between $\approx 16$ and $\approx 25$ over 2 orders of magnitude of model sizes for different language models. Compared to attacks on tokens, attacks on activations are predictably much stronger, however, we identify a surprising regularity where one bit of input steered either via activations or via tokens is able to exert control over a similar amount of output bits. This gives support for the hypothesis that adversarial attacks are a consequence of dimensionality mismatch between the input and output spaces. A practical implication of the ease of attacking language model activations instead of tokens is for multi-modal and selected retrieval models, where additional data sources are added as activations directly, sidestepping the tokenized input. This opens up a new, broad attack surface. By using language models as a controllable test-bed to study adversarial attacks, we were able to experiment with input-output dimensions that are inaccessible in computer vision, especially where the output dimension dominates.
BibTeX
@misc{fort2023scaling,
  author = {Stanislav Fort},
  title = {{Scaling Laws for Adversarial Attacks on Language Model Activations}},
  year = {2023},
  eprint = {2312.02780},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.02780},
}
Figure