Extracting Unlearned Information from LLMs with Activation Steering

Atakan Seyitoğlu, Aleksei Kuvshinov, Leo Schwinn, Stephan Günnemann

TL;DR

Supposedly "deleted" memories in unlearned Large Language Models (LLMs) can be recovered using Anonymized Activation Steering (AnonAct). By steering hidden activations with vectors derived from anonymized prompt contrasts, attackers can not only elicit forgotten information but also pinpoint the exact correct answer using simple token frequency. However, this attack works well for richly interconnected domains (e.g., the Harry Potter universe) while failing on isolated, narrow facts (e.g., fictitious author bios).

What Problem Does This Paper Address?

When LLMs memorize copyrighted or private training data, retraining from scratch is computationally infeasible. Instead, developers rely on machine unlearning to selectively remove targeted concepts or individuals (complying with privacy regulations like GDPR's "Right to Erasure").

While prior research has shown that unlearning can be leaky, existing extraction attacks suffer from a critical flaw: they produce large candidate sets containing the true answer, but give the attacker no reliable way to pinpoint which candidate is correct. Without an internal scoring mechanism or prior knowledge of the target data, evaluating real-world leakage risk remains inaccurate.

Methods

The authors propose Anonymized Activation (AnonAct) Steering, a representation-level intervention designed to bias the model back toward the deleted domain:

  1. Prompt Anonymization: For a domain-specific question $Q$ (e.g., "Who is the ghost of Gryffindor house in the Harry Potter series?"), the authors generate multiple domain-agnostic counterpart prompts $Q^*$ (e.g., "Who is the ghost of Braveheart house in the Caleb Mitchell series?"), replacing entities manually or using GPT-4.
  2. Steering Vector Extraction: The model processes both the original and anonymized questions. The authors extract hidden layer activations at the first generated token and compute a difference vector: $$S_l(Q) = A_l(Q) - \frac{1}{N}\sum_{n=1}^N A_l(Q^*_n)$$ Averaging these differences isolates the latent direction corresponding to the unlearned domain.

  3. Inference Steering: During decoding, this steering vector is added back to the model’s internal activations (scaled by a constant) solely at the first token.

  4. Answer Pinpointing via Word Frequency: The model generates 2,000 samples per query at a high sampling temperature ($T=2$, $\text{Top-}k=40$). The likelihood of an answer being correct is estimated by its maximum non-stopword frequency. If steering successfully boosts the correct answer frequency (CAF) above other noise, the most frequent candidate reveals the deleted fact.

Datasets

  • Curated Harry Potter Q&A: A newly introduced benchmark of 62 factual, keyword-based questions curated with GPT-4 (extending Schwinn et al., 2024), specifically selected because base Llama-2-7B answers them easily.
  • TOFU Benchmark (Maini et al., 2024): 40 questions covering two fictitious authors, converted from open-ended responses into keyword-based Q&A. Evaluated on an unlearned Phi-1.5 model.
  • CounterFact Dataset (Meng et al., 2022): 20 keyword-based single-token fact-replacement prompts used to evaluate ROME on GPT2-XL.

Key Insights and Ideas

  • Unlearning broad media universes $\neq$ unlearning isolated facts:
  • In a broad domain like Harry Potter, knowledge is redundant and deeply entangled across multiple characters, locations, and narrative arcs. Unlearning methods primarily sever surface associations rather than erasing underlying representations. AnonAct successfully restores these latent conceptual paths.
  • In contrast, narrow facts (e.g., an author's birthplace in TOFU, or a localized edit in ROME) rely on isolated training sentences. When that link is unlearned or edited, there are no latent associative pathways left to revive.
  • The first token dictates the trajectory: Aligning with prior mechanistic interpretability findings, steering internal activations exclusively during the generation of the very first token is sufficient to shift the probability mass across multi-token completions.

Key Results

  • Harry Potter Extraction:
  • The baseline unlearned model (WhoIsHarryPotter) achieved an Area Under the ROC Curve (AUC) of 0.75 (where 1.0 represents perfect retrieval via word frequency).
  • Applying AnonAct boosted the extraction AUC to 0.92, approaching the base Llama-2-7B model's performance of 0.98.
  • Targeted Unlearning (TOFU & ROME):
  • On TOFU, AnonAct failed to generalize; the correct answer frequency remained flat or degraded across virtually all queries.
  • On ROME (fact replacement), AnonAct pushed the prediction away from the artificial replacement token toward a uniform distribution, but failed to reconstruct the ground truth.

Limitations and Future Work

  • Failure on Specific/Narrow Facts: AnonAct struggles when facts do not share rich conceptual networks across the training distribution (e.g., non-public individuals or single-sentence factual edits).
  • Dependence on Paired Anonymizations: The attack requires generating plausible anonymized counter-prompts to compute effective steering vectors.
  • Future Directions: Developing unlearning schemes capable of thoroughly unlinking deeply entangled, broad-domain concepts, as well as designing more versatile extraction methods for granular, isolated knowledge.

Summary updated

Abstract

An unintended consequence of the vast pretraining of Large Language Models (LLMs) is the verbatim memorization of fragments of their training data, which may contain sensitive or copyrighted information. In recent years, unlearning has emerged as a solution to effectively remove sensitive knowledge from models after training. Yet, recent work has shown that supposedly deleted information can still be extracted by malicious actors through various attacks. Still, current attacks retrieve sets of possible candidate generations and are unable to pinpoint the output that contains the actual target information.We propose activation steering as a method for exact information retrieval from unlearned LLMs. We introduce a novel approach to generating steering vectors, named Anonymized Activation Steering. Additionally, we develop a simple word frequency method to pinpoint the correct answer among a set of candidates when retrieving unlearned information. Our evaluation across multiple unlearning techniques and datasets demonstrates that activation steering successfully recovers general knowledge (e.g., widely known fictional characters) while revealing limitations in retrieving specific information (e.g., details about non-public individuals). Overall, our results demonstrate that exact information retrieval from unlearned models is possible, highlighting a severe vulnerability of current unlearning techniques.
BibTeX
@inproceedings{seyitoglu2024extracting,
  author = {Atakan Seyitoğlu and Aleksei Kuvshinov and Leo Schwinn and Stephan Günnemann},
  title = {{Extracting Unlearned Information from LLMs with Activation Steering}},
  booktitle = {SafeGenAi @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=RuufZiUWUq},
}
Figure