Decomposing how prompting steers behavior

Fan L. Cheng, Nikolaus KriegeskortearXiv 2026 · arXiv

TL;DR

Prompting changes the behavior of large language and vision-language models without updating their weights, but how this happens geometrically inside the network has been a black box. This paper introduces a nested geometric framework to show that while simple vector translations explain most of the activation variance, affine transformations (cross-dimensional linear feature mixing) are required to actually restructure internal geometry and causally steer downstream behavior.

What Problem Does This Paper Address?

When you change a prompt from "Is there a person in this image?" to "How many people are in this image?", the model completely shifts its behavior from binary classification to counting. Because this happens entirely during inference without weight updates, the prompt must be steering the model by reshaping its internal activation space (representational geometry).

Prior work has shown that prompts alter internal representations, but usually relies on simple scalar metrics or basic activation steering (like adding a single "steering vector"). We still lack a rigorous understanding of the exact mathematical class of transformations that natural prompts induce. Does a prompt simply shift the data cloud, rotate it, scale it, or perform complex feature mixing? This paper fills that gap by systematically decomposing prompt-induced changes into a hierarchy of interpretable geometric operations.

Methods

The authors propose a nested geometric decomposition framework. For any pair of prompts (Prompt A and Prompt B) acting on the same set of stimuli, they fit a sequence of increasingly expressive, stimulus-invariant mathematical maps to align the model's hidden states from Prompt A to Prompt B:

  1. Translation ($F_T$): A simple mean-shift (adding a constant vector).
  2. Rigid + Uniform Scaling ($F_{Ou}$): Global rotation/reflection plus uniform scaling (Procrustes alignment).
  3. Rigid + Axis-wise Scaling ($F_{Oa}$): Procrustes alignment followed by independent scaling of each aligned axis (gain modulation).
  4. Affine ($F_L$): Full linear feature mixing plus translation (shearing and stretching).
  5. Nonlinear ($F_N$): A multi-layer perceptron (MLP) to capture non-linear warping.
Experimental setup for mapping representational manifolds between different prompts.

To prove these maps aren't just fitting noise, the authors perform causal interventions: they feed a new stimulus using Prompt A, stop the model at a specific layer $\ell$, apply the fitted transformation to the hidden state, and let the model finish its forward pass. They then evaluate whether the model's internal geometry and final text outputs successfully match what Prompt B would have produced.

Datasets

The framework was evaluated across three LLMs (OPT-2.7B, Llama3-8B, Qwen3-8B) and three VLMs (BLIP-2, LLaVA-OneVision, Qwen3-VL) using six datasets structured factorially with two main attributes:

  • Text Datasets (for LLMs): EmotionalStory (topic vs. emotion), WritingStyle (topic vs. style), and Number (numerical properties vs. cognitive operations).
  • Image Datasets (for VLMs): EmoSet (image content vs. evoked emotion), StyleTransfer (scene content vs. artistic style), and COCO (person detection vs. person counting).

Key Insights and Ideas

  • Prompts Actively Reorganize Task Geometry: Representational Similarity Analysis (RSA) reveals that prompts warp the representation space to cluster stimuli based on the instructed task. Under a counting prompt, the internal representations arrange themselves in an ordinal, count-graded block structure that is completely absent under a detection prompt.
Figure 2: Prompting reshapes representational geometry toward the instructed task structure. (a) Multidimensional scaling (MDS) visualizations of prompt-conditioned representations. (Top) Layer-32 representations from Llama3-8B-Instruct for 1,920 text stories under a topic prompt (Prompt A) and an emotion prompt (Prompt B). (Bottom) Layer-27 representations from LLaVA-OneVision for 1,000 COCO images under a person-detection prompt (Prompt A) and a person-counting prompt (Prompt B). Each point denotes one stimulus and is colored by the target label for Prompt B.(b) Representational dissimilarity matrix (RDM) for the same model, layer, and image set as in the bottom row of panel (a). The top two RDMs are computed from the prompt-A and prompt-B hidden states, respectively. The bottom RDM is the target task RDM for prompt B, constructed from numerical distances between count labels. (c) Layerwise alignment with the target task structure (pooled across models and datasets). Left: Spearman correlation between each prompt-induced RDM and the corresponding target RDM, computed separately for prompt A and prompt B. Right: silhouette score using the target labels, which measures whether stimuli with the same target label form compact, well-separated clusters in representation space. The shaded regions show ±1 SD.
  • The "Affine" Threshold for Behavior: While pure translation ($F_T$) accounts for the largest absolute share of variance in activation changes, it is insufficient for behavioral steering. The affine tier ($F_L$) is the critical threshold: it is the first class of transformation that nearly fully recovers the target prompt's task geometry and unlocks massive gains in behavioral accuracy. This implies that cross-dimensional linear feature mixing is the primary mechanism for task routing.
  • Model-Specific Routing Strategies: Different model families use different internal routing mechanisms. Modern instruction-tuned models (like Llama3 and Qwen3) rely heavily on translation in early layers, suggesting they use a low-dimensional "instruction code." Conversely, older base models like OPT-2.7B rely more on distributed, higher-dimensional rotational and affine adjustments spread across intermediate layers.

Key Results

  • Monotonic Growth with Depth: The magnitude of prompt-induced representational change is tiny in early layers but grows monotonically with depth, peaking in the final layers of the models.
  • Variance Explained: Across almost all models and datasets, the hierarchy of incremental variance explained follows the strict order of $F_T > F_L > F_{Ou} > F_{Oa} \approx F_N$.
  • Causal Recovery: In causal text generation tests, moving from a translation map to an affine map consistently shifts the model's output format to match the target task (e.g., successfully forcing a model primed on an irrelevant factual prompt to output image-dependent counts). Adding non-linear capabilities ($F_N$) yields negligible improvements over the affine map, proving that the population-level effect of prompting is fundamentally affine.

Limitations and Future Work

The authors note several boundaries to the current study:

  1. Scope of Prompting: The analysis is strictly limited to zero-shot instruction prompts. It does not yet cover few-shot demonstrations, soft prompts, or long-context chain-of-thought conditioning.
  2. Intervention Granularity: The causal interventions replace the residual stream at a single layer and only at the final input-prompt token.

Future work will explore multi-layer and multi-token interventions, as well as testing how robustly these geometric maps generalize under severe input-distribution shifts.

Summary updated

Abstract

Prompting steers large language models (LLMs) and vision-language models (VLMs) without weight updates, but it remains unclear how instruction changes reshape internal representations to produce behavior. We introduce a nested geometric decomposition framework that treats prompting as a transformation of the representational geometry of the content following the prompt. For each prompt pair, we align representations of the same stimuli under two prompts using increasingly expressive stimulus-invariant maps: translation, rigid transformation with uniform scaling, sequential axis scaling, affine transformation, and nonlinear transformation. We then causally test each map by replacing a single layer's prompt-A hidden state for held-out stimuli with its mapped counterpart and measuring recovery of prompt-B representational geometry and behavior. Across three LLMs, three VLMs, and six text or image datasets spanning style, emotion, scene content, and number, prompts consistently reshape representations toward the instructed task structure. Cross-validated variance decomposition shows that much prompt-induced activation change is captured by shape-preserving maps, especially translation and rigid transformation with uniform scaling, while tier profiles reveal model- and task-specific routing strategies across layers. Crucially, although translation and rigid tiers already improve behavioral agreement, affine transformation is the first tier to nearly recover target-prompt task geometry and yields corresponding behavioral gains. This suggests that cross-dimensional linear mixing is a key mechanism by which prompts reorganize representations toward instructed task structure. Our framework decomposes prompt-induced representational change into interpretable geometric components and reveals how models route task-relevant structure to produce prompt-driven behavior.
BibTeX
@misc{cheng2026decomposing,
  author = {Fan L. Cheng and Nikolaus Kriegeskorte},
  title = {{Decomposing how prompting steers behavior}},
  year = {2026},
  eprint = {2606.03093},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.03093},
}

Loading related content...

Figure