Extracting Latent Steering Vectors from Pretrained Language Models

Nishant Subramani, Nivedita Suresh, Matthew E. PetersACL 2022 · DOI

Introduction

Controllable text generation—the ability to guide a model's output toward specific attributes like sentiment, topic, or style—is a central challenge in natural language processing. Traditional methods often rely on computationally expensive fine-tuning, complex prompt engineering, or the training of auxiliary "plug-and-play" modules. This research explores a more fundamental hypothesis: the information required to generate any specific sentence is already encoded within the frozen weights of a pretrained language model. Instead of modifying the model, the authors demonstrate that one can simply find the correct "nudge" in the model's internal activations to produce a desired output, uncovering a rich "latent steering space."

Methodology

The primary contribution of this work is the extraction and application of a steering vector ($z_{steer}$). This process involves three key stages:

  1. Optimization: For a target sentence, a fixed-length vector is randomly initialized. Using the Adam optimizer, this vector is updated via gradient descent to maximize the likelihood of the model generating the target sentence, while the language model's weights remain strictly frozen.
  2. Injection: The optimized vector is added to the model's hidden states during the forward pass. The researchers investigated various injection points, including different layers and timesteps (e.g., injecting only at the first token versus every token).
  3. Vector Arithmetic: To enable unsupervised tasks like sentiment transfer, the authors utilized the geometric properties of the latent space. By calculating "offset vectors" (the difference between average positive and negative steering vectors), they could "flip" the sentiment of a sentence by simply adding the offset to its original steering vector.
Architecture of steering vector injection into a transformer model.

Experimental Setup

The approach was validated across a diverse set of benchmarks:

  • Sentence Recovery: Evaluated on a multi-domain corpus including Cornell Movie Dialogs, Project Gutenberg, Gigaword, and WikiText-103.
  • Sentiment Transfer: Tested on the Yelp sentiment benchmark to compare against specialized autoencoder models.
  • Semantic Similarity: Measured using the STS-B (Semantic Textual Similarity Benchmark).
  • Style Transfer: Evaluated on StylePTB for fine-grained structural changes, such as active-to-passive voice conversion.

Results and Discussion

The "Goldilocks" Layers

A critical finding was the sensitivity of the model to the injection location. Steering is most effective when the vector is injected into the middle layers (specifically layers 6 or 7 of a 12-layer GPT-2). Injecting at the embedding layer lacks the necessary complexity, while injecting at the final output head provides insufficient processing depth for the model to integrate the steering signal effectively.

Spearman correlation on STS-B across different injection layers, comparing feed-forward and self-attention blocks against baselines.

Performance Benchmarks

  • Near-Perfect Recovery: By targeting middle-layer injections, the authors achieved a BLEU-4 score exceeding 99, indicating nearly perfect reconstruction of target sentences from a single vector.
  • Semantic Superiority: On the STS-B benchmark, steering vectors achieved a Spearman correlation of approximately 52. This significantly outperforms standard extractive methods, such as mean-pooling GPT-2 (25.9) or BERT (47.3) hidden states, suggesting that the optimization process captures deeper semantic nuances.
  • Latent Space Continuity: Linear interpolation between steering vectors resulted in grammatical sentences that smoothly blended the meanings of the source sentences, proving the latent space is continuous and well-formed.

Beyond Memorization

To ensure the model wasn't simply memorizing strings, the authors attempted to steer the model toward "gibberish" and shuffled text. They found it significantly harder to generate nonsense than natural language, confirming that steering vectors leverage the model's intrinsic linguistic knowledge rather than overriding it.

Limitations and Future Work

Despite its efficacy, the method has drawbacks. The optimization process is computationally intensive, requiring a gradient descent loop for every new sentence. Additionally, while the latent space is smooth, sampling novel, coherent sentences from scratch remains difficult; only about 20% of randomly sampled vectors produce valid text. Future research may focus on training "super-encoders" to predict these steering vectors in a single forward pass, potentially enabling real-time, high-fidelity control of any frozen language model.

Summary updated

BibTeX
@inproceedings{subramani2022extracting,
  author = {Nishant Subramani and Nivedita Suresh and Matthew E. Peters},
  title = {{Extracting Latent Steering Vectors from Pretrained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.48},
}
Figure