How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via Mechanistic Interpretability

Jorge García-Carrasco, Alejandro Maté, Juan C. TrujilloAISTATS 2024

Abstract

Mechanistic Interpretability (MI) aims to reverse-engineer the internal computations of Large Language Models (LLMs). While previous research has largely focused on single-token prediction tasks, this study investigates a more complex, multi-token behavior: the prediction of three-letter acronyms (e.g., "CEO" from "Chief Executive Officer"). By employing activation patching and circuit extraction techniques on GPT-2 Small, we identify a specific "circuit" of just eight attention heads responsible for this task. Our findings reveal a specialized class of "Letter Mover Heads" and a novel reliance on the "Beginning of Sequence" (BOS) token as a positional heuristic.

Introduction

Large Language Models (LLMs) are often described as "black boxes" due to the opacity of their internal decision-making processes. This lack of transparency poses significant challenges for AI safety and alignment. Mechanistic Interpretability (MI) seeks to address this by identifying the specific sub-networks, or "circuits," that perform particular tasks.

Most existing MI research has focused on simple, single-token tasks, such as identifying the indirect object in a sentence. This paper extends this work to a multi-stage behavior: predicting a sequence of tokens to form an acronym. This task requires the model to extract the first letter of multiple words and maintain their order over several prediction steps.

Methods

To analyze GPT-2 Small (117M parameters), the researchers utilized several key MI techniques:

  1. Activation Patching: This technique involves "corrupting" the input (e.g., replacing "Chief" with a random word) and then systematically restoring internal activations to their original state to see which ones recover the model's performance. This helps pinpoint where relevant information is processed.
  2. Circuit Extraction: By isolating the components identified through patching, the researchers defined a sub-network consisting of only 8 attention heads.
  3. Ablation Studies: To validate the circuit, the researchers "muted" all other parts of the model (mean ablation) to determine if the 8-head circuit could perform the task in isolation.
Residual stream patching across sequence positions for the first, second, and third letters of an acronym.

The patching results in Figure 1 demonstrate that the model relies on activations at the positions of the words being acronymized (C1, C2, C3) to predict the corresponding letters at different layers of the residual stream.

Key Insights: The Acronym Circuit

The investigation revealed a highly specialized mechanism within the model.

Letter Mover Heads

The core of the circuit consists of "Letter Mover Heads" (specifically Heads 8.11, 10.10, 9.9, and 11.4). These heads function as translators, extracting the initial capital letter from a word and writing it into the residual stream at the current prediction position.

Attention head patching results identifying the specific heads responsible for predicting each letter of the acronym.

As shown in Figure 5, these heads are concentrated in the later layers of the model, particularly layers 8 through 11, where they integrate information to form the final prediction.

Positional Heuristics and the BOS-Effect

A surprising finding was the model's use of the "BOS-effect" rather than standard positional embeddings. Because of the causal mask in transformers, attention to the "Beginning of Sequence" (BOS) token naturally decreases as the sequence length increases. The model uses this decaying attention as a proxy for distance, allowing it to determine which word's first letter should be extracted next in the sequence.

Key Results

The study yielded several significant findings:

  • Efficiency: A remarkably small portion of the model—just 8 out of 144 attention heads (~5%)—is responsible for the acronym-prediction capability.
  • Performance: When isolated, this 8-head circuit achieved a 90.2% probability of picking the correct letter, maintaining a logit difference of approximately 2.22.
  • Generalization: The circuit proved robust across various prompts, suggesting it represents a generalized algorithmic structure rather than a memorized pattern.

Discussion and Future Work

While this study successfully reverse-engineered a multi-token task in GPT-2 Small, several questions remain. It is unclear if such circuits remain as localized in larger, more modern models like GPT-4. Additionally, the acronym task is highly structured; applying these techniques to more "fuzzy" tasks like creative writing or complex logical reasoning remains a frontier for future research. Finally, the role of "negative heads"—which appear to suppress overconfidence—warrants further investigation to fully understand the model's calibration.

Summary updated

BibTeX
@inproceedings{garciacarrasco2024how,
  author = {Jorge García-Carrasco and Alejandro Maté and Juan C. Trujillo},
  title = {{How does GPT-2 Predict Acronyms? Extracting and Understanding a Circuit via Mechanistic Interpretability}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
}
Figure