Spectral Editing of Activations for Large Language Model Alignment

Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo M. Ponti, Shay B. Cohen

TL;DR

Spectral Editing of Activations (SEA) is a training-free, inference-time method that aligns Large Language Models (LLMs) by surgically modifying their internal representations. By using spectral decomposition to project activations toward "truthful" directions and away from "biased" ones, SEA significantly improves model honesty and fairness with minimal computational overhead and as few as 25 demonstrations.

What Problem Does This Paper Address?

Despite their impressive capabilities, LLMs frequently generate untruthful content (hallucinations) or reflect societal biases present in their training data. Traditional alignment techniques like Reinforcement Learning from Human Feedback (RLHF) or Supervised Fine-Tuning (SFT) are computationally expensive and often require massive datasets.

Recent research suggests that LLMs often "know" the truth internally even when they "say" something false. This implies that the information for alignment is already present in the model's activation space. The challenge lies in finding a way to steer these internal states during inference to ensure the output aligns with human preferences without the need for costly retraining or degrading the model's general reasoning abilities.

Methods

The authors introduce Spectral Editing of Activations (SEA), which operates in two distinct stages:

  1. Offline Projection Calculation: The researchers collect "neutral" activations from the model, along with activations from pairs of "positive" (e.g., truthful) and "negative" (e.g., hallucinated) demonstrations. They then compute the cross-covariance matrices between the neutral activations and the polarized ones. Using Singular Value Decomposition (SVD), they identify the principal directions in the activation space that correlate most with positive behavior and those that correlate with negative behavior.
  2. Inference-Time Editing: When a user queries the model, SEA intercepts the activations at specific layers (typically the final MLP outputs of Transformer blocks). It projects these activations to maximize covariance with the positive directions while minimizing it with the negative ones.
Figure 2: An overview of Spectral Editing of Activations (SEA). The method consists of two stages: (Left) the offline calculation of the editing projections using spectral decomposition with positive, negative and neutral demonstrations. (Right) the application of the calculated editing projections during LLM inference, thus manipulating predictions.

Recognizing that some behaviors (like bias) might not be linearly separable, the authors also propose $\Phi$-SEA. This extension uses an invertible non-linear feature function to map activations into a higher-dimensional space for editing before transforming them back.

Datasets

The method was evaluated across six open-source LLMs (LLaMA-2, Gemma, and Mistral families) using the following benchmarks:

  • Truthfulness: Evaluated on TruthfulQA (817 questions) using projections derived from HaluEval.
  • Bias/Fairness: Evaluated on the BBQ (Bias Benchmark for QA), covering 11 categories of common social biases.
  • Generalization: Tested on CrowS-Pairs to see if editing for BBQ generalizes to other bias datasets.
  • Control Tasks: To ensure general capabilities weren't harmed, they tested on MMLU, HellaSwag, Natural Questions, GSM8K, and MathQA.

Key Insights and Ideas

  • Linear vs. Non-linear Separability: The authors found that while truthfulness is often linearly separable in the activation space, social bias is more complex. As shown in the t-SNE plots below, positive and negative clusters for truthfulness (HaluEval) are distinct, whereas bias (BBQ) clusters overlap significantly, necessitating the non-linear $\Phi$-SEA approach.
Figure 1: t-SNE plot of LLaMA- 2-chat-7B’s activations for positive (blue) and negative (red) demonstra- tions from HaluEval and BBQ.
  • Data Efficiency: Unlike fine-tuning, which requires thousands of examples, SEA shows noticeable improvements with just 25 demonstrations.
  • Layer-wise Importance: Spectral analysis revealed that truthfulness information is typically concentrated in the top layers of the model, justifying why editing only the final layers is effective and efficient.
  • Minimal "Tax" on Reasoning: Linear SEA has almost zero impact on the model's ability to solve math problems or reason about commonsense, a common side effect of heavy fine-tuning.

Key Results

  • Truthfulness: On LLaMA-2-chat-7B, SEA improved the MC1 score on TruthfulQA from 36.96 to 39.41.
  • Bias Mitigation: The non-linear $\Phi$-SEA (using a squared-exponential kernel) was particularly potent, boosting BBQ accuracy from 43.02% to 56.17% and slashing the bias score from 5.8% to 2.0%.
  • Efficiency: SEA is incredibly fast. Calculating projections for 21 layers takes about 2.5 minutes on a single A100 GPU, and the inference-time overhead is a negligible ~3.7%.
Figure 3: Left: Accuracy of all methods on each group of bias type in BBQ. Right: results on BBQ’s testing set. All methods are applied on LLaMA-2-Chat-7B. For accuracy (A%↑), higher values are better; for unknown-answer response rate (U%↓), bias score (BS%↓) and stereotypical response rate (SR%↓), lower is better. We use bold font for the best result in each column, and mark the methods that improve ICL. †: significant improvements on ICL in A% by pair-wise t-test with p < 0.05.

Limitations and Future Work

The primary limitation identified is that non-linear editing ($\Phi$-SEA) can cause a slight performance drop in complex reasoning tasks like math. This is likely because the "pseudo-inverse" used to return activations to the original space is not perfectly lossless.

Future research directions include finding more theoretically robust non-linear transformations that preserve general capabilities better and exploring how to jointly edit for multiple attributes (e.g., being simultaneously more truthful and less biased) without them interfering with each other.

Summary updated

Abstract

Large language models (LLMs) often exhibit undesirable behaviours, such as generating untruthful or biased content. Editing their internal representations has been shown to be effective in mitigating such behaviours on top of the existing alignment methods. We propose a novel inference-time editing method, namely spectral editing of activations (SEA), to project the input representations into directions with maximal covariance with the positive demonstrations (e.g., truthful) while minimising covariance with the negative demonstrations (e.g., hallucinated). We also extend our method to non-linear editing using feature functions. We run extensive experiments on benchmarks concerning truthfulness and bias with six open-source LLMs of different sizes and model families. The results demonstrate the superiority of SEA in effectiveness, generalisation to similar tasks, as well as computation and data efficiency. We also show that SEA editing only has a limited negative impact on other model capabilities.
BibTeX
@inproceedings{qiu2024spectral,
  author = {Yifu Qiu and Zheng Zhao and Yftah Ziser and Anna Korhonen and Edoardo M. Ponti and Shay B. Cohen},
  title = {{Spectral Editing of Activations for Large Language Model Alignment}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=pqYceEa87j},
}
Figure