Scaling Laws for Adversarial Attacks on Language Model Activations and Tokens

Stanislav FortICLR 2025 · OpenReview

Introduction

Adversarial attacks—small, intentional perturbations that cause a machine learning model to malfunction—have been studied extensively in computer vision. However, their behavior in Large Language Models (LLMs) is more complex due to the discrete nature of text. This paper investigates a fundamental question: How much control does an attacker actually have over an LLM's output?

Specifically, the research explores the "dimensionality mismatch" hypothesis, which suggests that models are vulnerable because their input space (high-dimensional continuous vectors) is vastly larger than their output space (discrete predicted tokens). By treating LLMs as a controllable test-bed, the authors quantify this vulnerability and derive a linear scaling law: the number of output tokens an attacker can precisely dictate grows linearly with the number of input activations they control.

Attack Methodologies

The researchers compared two primary attack vectors to understand the limits of adversarial control:

  1. Activation Attacks: The attacker injects a continuous perturbation vector directly into the model's internal activations (the "residual stream") immediately following the embedding layer. Because these activations are continuous floating-point numbers, the attacker can use gradient-based optimization (like Adam) to find the exact "nudge" needed to force a specific output sequence.
Example of an activation attack where a perturbation vector is added to the first token to force an unrelated output.
  1. Token Substitution Attacks: A more traditional, discrete approach where the attacker replaces input tokens with others. The authors used a greedy, exhaustive search over the vocabulary to find the optimal tokens to achieve the target output.

To probe these limits further, the study introduced "multi-attacks" (forcing one perturbation to solve multiple tasks simultaneously) and "fractional dimension attacks" (restricting the attacker to only change a subset of the vector's dimensions).

Experimental Setup

The study utilized several well-known open-source model families to ensure the findings generalized across architectures:

  • EleutherAI/Pythia: Ranging from 70M to 2.8B parameters.
  • Microsoft/Phi-1 and Phi-1.5: High-performing small language models (1.3B parameters).
  • TinyStories-33M: A small model trained on simple synthetic stories.

Attacks were tested using both randomly sampled tokens and natural language sequences, such as forcing a math problem prompt to output a specific poem.

Scaling Laws and Attack Resistance

The core contribution of this paper is the empirical verification of scaling laws governing these attacks.

  • The Linear Scaling Law: The maximum number of target tokens ($t_{max}$) an attacker can dictate follows a simple linear relationship: $t_{max} = \kappa a$, where $a$ is the number of tokens whose activations the attacker controls, and $\kappa$ is the "attack multiplier."
Attack success drops as target length increases, but the maximum target length scales linearly with the attack length.
  • Attack Resistance ($\chi$): The authors define attack resistance as the number of bits of input control needed to exert one bit of control over the output. Surprisingly, this value is remarkably stable (between $\approx 16$ and $25$) across different model architectures and sizes.
  • Activations vs. Tokens: Activation attacks are predictably much stronger than token attacks. For instance, in a Pythia-70M model, controlling the activations of a single token allows an attacker to dictate ~24 output tokens. Conversely, it takes ~8 input tokens to dictate just one output token via discrete substitution.
  • Dimensionality Mismatch: Despite the raw power difference, the efficiency of the control (bits of input vs. bits of output) is surprisingly similar between activation and token attacks. This strongly supports the theory that adversarial vulnerability is a direct consequence of the high dimensionality of the model's internal representations.

Key Results and Implications

The level of control demonstrated is massive. By manipulating all dimensions of a single token's activation, the authors successfully forced a model to recite a specific 40-token poem as the answer to "1 + 2 = ". Furthermore, these attacks exhibit "distance robustness," remaining effective even with a gap of up to 100 tokens between the manipulated input and the target output.

While activation attacks require "white-box" access (knowledge of internal weights and the ability to inject vectors), this poses a severe, practical risk for multi-modal models (where continuous image embeddings are converted to activations) and Retrieval-Augmented Generation (RAG) systems. In these setups, attackers might control the continuous vectors injected into the prompt, bypassing the discrete token bottleneck entirely. Future work will need to investigate how defense mechanisms, quantization, or safety training might alter a model's inherent attack resistance.

Summary updated

Abstract

We explore a class of adversarial attacks targeting the activations of language models to derive upper-bound scaling laws on their attack susceptibility. By manipulating a relatively small subset of model activations, $a$, we demonstrate the ability to control the exact prediction of a significant number (in some cases up to 1000) of subsequent tokens $t$. We empirically verify a scaling law where the maximum number of target tokens predicted, $t_\mathrm{max}$, depends linearly on the number of tokens $a$ whose activations the attacker controls as $t_\mathrm{max} = \kappa a$. We find that the number of bits the attacker controls on the input to exert a single bit of control on the output (a property we call \textit{attack resistance $\chi$}) is remarkably stable between $\approx 16$ and $\approx 25$ over orders of magnitude of model sizes and between model families. Compared to attacks directly on input tokens, attacks on activations are predictably much stronger, however, we identify a surprising regularity where one bit of input steered either via activations or via tokens is able to exert a surprisingly similar amount of control over the model predictions. This gives support for the hypothesis that adversarial attacks are a consequence of dimensionality mismatch between the input and output spaces. A practical implication of the ease of attacking language model activations instead of tokens is for multi-modal and selected retrieval models. By using language models as a controllable test-bed to study adversarial attacks, we explored input-output dimension regimes that are inaccessible in computer vision and greatly extended the empirical support for the dimensionality theory of adversarial attacks.
BibTeX
@inproceedings{fort2025scaling,
  author = {Stanislav Fort},
  title = {{Scaling Laws for Adversarial Attacks on Language Model Activations and Tokens}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=YzxMu1asQi},
}
Figure