Scaling Effects on Latent Representation Edits in GPT Models (Student Abstract)

Austin L. Davis, Gita SukthankarAAAI 2025 · DOI

TL;DR

This paper explores how to effectively "edit" the internal workings of a GPT model trained on chess moves by manipulating its hidden states. They found that the strength of these edits needs to be carefully scaled, following a negative exponential relationship with the input sequence length, to maintain the model's ability to generate valid chess moves. This research offers a principled way to control and understand the emergent representations within large language models.

What Problem Does This Paper Address?

Large language models (LLMs) like GPTs learn complex representations of their training data. Researchers are keen to understand what these internal representations "mean" and how they can be manipulated to control the model's behavior. One approach is to use "probing classifiers" – smaller models trained to predict specific information from an LLM's internal states. This paper builds on this by investigating how to edit these internal states to influence the LLM's output. The challenge is that arbitrary edits can easily break the model's understanding, leading to nonsensical or invalid outputs. The authors specifically address how to scale these edits to ensure semantic validity, using the highly structured domain of chess as a testbed.

Methods

The researchers trained a 12-layer GPT-2 model on UCI-encoded chess move sequences. They then trained a linear probe classifier to predict the chess board state from the GPT's intermediate activations at a specific layer. To perform edits, they identified a target piece and square to remove (or add) and used the corresponding column vector from the probe's weight matrix as an "intervention vector." This vector was then added to the GPT's residual stream activation. The crucial part of their method is the scaling of this intervention vector. They introduced a scaling factor, denoted by $\eta$, which is multiplied by the intervention vector. The core of their investigation is to determine the optimal way to set $\eta$ to ensure the GPT's subsequent move predictions remain valid. They systematically varied $\eta$ and measured the legality of the GPT's output.

Datasets

The paper used a dataset of 120,000 chess games encoded in UCI format for training the GPT and for generating activation vectors for probe training. The evaluation was performed on a hold-out set of 500 random games. No new benchmarks were introduced; existing chess game datasets were utilized.

Key Insights and Ideas

  • Chess as a Controlled Environment: Using chess, a domain with strict logical rules, allows for a clear, objective measure of "semantic validity" (i.e., move legality), which is difficult to define for natural language.
  • Intervention Vector Scaling is Crucial: Simply adding an intervention vector to the residual stream is not enough. The magnitude of this intervention ($\eta$) significantly impacts the validity of the model's output.
  • Negative Exponential Scaling: The optimal scaling factor $\eta$ appears to follow a negative exponential decay with respect to the input sequence length (move number). This suggests that as the model processes more of the sequence, the influence of earlier activations diminishes, and interventions need to be adjusted accordingly.
  • "Goldilocks" Region for $\eta$: There's a narrow range of $\eta$ values that leads to successful interventions. Too little scaling has no effect, while too much scaling can corrupt the model's internal representation, leading to invalid moves.
  • Hypothesized Reason for Decay: The authors hypothesize that the residual stream's information becomes more nuanced later in a sequence. Large $\eta$ values risk overwriting this nuance, corrupting the model's internal world model.

Key Results

  • The GPT model achieved a high accuracy (99.9%) in recommending legal moves on a hold-out evaluation dataset without any interventions.
  • The linear probe classifier achieved an average F1-score exceeding 0.90, with scores above 0.94 on layers 7 and 8.
  • When an appropriate $\eta$ was chosen (around 0.3 for many moves), interventions successfully altered the GPT's output to make illegal moves invalid 92% of the time (with a standard deviation of 0.02).
  • The intervention's effectiveness is highly sensitive to $\eta$ values outside a specific "Goldilocks" region, which varies across moves.
  • For the first move, a larger $\eta$ is required for effectiveness, and over-scaling has fewer consequences compared to later moves.

Limitations and Future Work

  • Domain Specificity: The current research is limited to GPTs trained on chess. The authors acknowledge that future work will investigate these scaling effects on GPTs trained on natural language, where defining "semantic validity" is more challenging.
  • Generalizability of Scaling Rule: While a negative exponential scaling rule is proposed, its generalizability to different model architectures, training data, and intervention types needs further exploration.
  • Understanding Nuance: The hypothesis that later sequence information is more nuanced and susceptible to corruption by large interventions is an area for deeper investigation.

The authors suggest that understanding and controlling these latent space edits has significant implications for controlling the emergent representations of GPTs trained on various tasks.

Summary updated

BibTeX
@inproceedings{davis2025scaling,
  author = {Austin L. Davis and Gita Sukthankar},
  title = {{Scaling Effects on Latent Representation Edits in GPT Models (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35245},
}
Figure