Soft Token Attacks Cannot Reliably Audit Unlearning in Large Language Models

Haokun Chen, Sebastian Szyller, Weilin Xu, Nageen Himayat

TL;DR

This paper demonstrates that soft token attacks (STAs), a popular method for auditing machine unlearning in large language models (LLMs), are unreliable. The authors show that STAs can extract information even from models that have supposedly undergone unlearning, and can even generate random strings, rendering them ineffective for verifying true unlearning.

What Problem Does This Paper Address?

Large language models (LLMs) are trained on massive datasets that can contain sensitive or undesirable information, such as personal data, copyrighted material, or harmful content. Machine unlearning aims to remove this specific information from trained LLMs without retraining them from scratch. However, verifying that unlearning has been successful is challenging. Recent work proposed "soft token attacks" (STAs) as a way to audit unlearning by attempting to elicit the "unlearned" information. This paper questions the effectiveness of STAs for this purpose, particularly in a "strong auditor" setting where the auditor has significant capabilities.

Methods

The authors propose a framework for auditing unlearning using soft token attacks (STAs). Their approach involves an "auditor" that uses STAs to probe an unlearned model. The core idea is that if unlearning is successful, it should be significantly harder to elicit specific information from the unlearned model compared to the original, fine-tuned model.

Specifically, they consider two types of auditors:

  • Oracle Auditor (A₀): This auditor has perfect knowledge of the training data and can definitively determine if information was present. This is an idealized scenario.
  • STA Auditor (ASTA): This auditor uses soft token attacks. They optimize an "adversarial suffix" (xs) that, when combined with a base prompt (xp), aims to elicit a target completion (c). The key difference from prior work is that ASTA optimizes xs with respect to the entire target completion, not just a part of it, representing a more powerful auditing scenario.

They evaluate STAs on two common unlearning benchmarks: "Who Is Harry Potter?" (WHP) and "TOFU."

Datasets

  • Who Is Harry Potter? (WHP): This dataset aims to remove information about the Harry Potter universe. The authors used snippets from its Hugging Face page and augmented it with Llama2-generated trivia pairs.
  • TOFU: This dataset is designed to remove information about fictional writers, intended to be absent from LLM training data. The authors used a 10% forget/90% retain split.

Key Insights and Ideas

  • STAs are too powerful for auditing: The primary insight is that STAs, especially in a strong auditor setting, are too effective at eliciting information. They can retrieve content that should have been unlearned, and even generate seemingly random strings, making it impossible to distinguish between a truly unlearned model and one that simply has a weak unlearning algorithm applied.
  • Unlearning algorithms are vulnerable: The paper suggests that current approximate unlearning methods are not robust enough to withstand sophisticated auditing techniques like STAs.
  • Random string generation is a red flag: The ability of STAs to generate long, random strings is particularly concerning. This implies that the attack is not targeting specific learned knowledge but rather exploiting the model's generative capabilities in a way that bypasses the unlearning process.
  • Benchmark limitations: The "Who Is Harry Potter?" dataset is questioned as a reliable benchmark due to variations in prompt templates and the lack of a standardized version.

Key Results

  • Eliciting unlearned information: In a strong auditor setting, STAs could elicit target completions from unlearned models across all tested unlearning methods and benchmark datasets. This held true even for base models not fine-tuned on the specific benchmark data.
  • Random string generation: A single soft token could elicit approximately 150 random characters, and ten soft tokens could generate over 400 random characters. This demonstrates the attack's ability to produce arbitrary output, not just targeted information.
  • Number of soft tokens: For the WHP dataset, approximately 4-6 soft tokens were sufficient to elicit completions. For TOFU, around 3 soft tokens were needed. The paper notes that with increased iterations, even 1-2 soft tokens could elicit completions.
  • No significant difference: For most pairs of models (base vs. unlearned), statistical tests (Welch's t-test) showed no significant evidence that eliciting completions was more difficult after unlearning.

Limitations and Future Work

  • Model Size: The experiments were conducted on models with 7-8 billion parameters. The authors expect the findings to extend to larger models due to the general scaling of LLM capabilities.
  • Efficiency: The evaluation was limited to at most 10 soft tokens and random strings up to 400 characters. Determining the upper bounds for these parameters is a future direction.
  • Black-box Optimization: The current STA implementation assumes white-box access. Future work could explore whether black-box optimization methods can achieve similar elicitation results.
  • Understanding Learned Tokens: Investigating what specific soft tokens are learned during the auditing process could provide deeper insights.
  • Ethical Considerations: The paper highlights that auditors with sufficient resources and knowledge can elicit arbitrary (potentially harmful) information from LLMs, even with approximate prior knowledge of the removed content. This underscores the need for more robust unlearning and auditing mechanisms.

Summary updated

Abstract

Large language models (LLMs) are trained using massive datasets, which often contain undesirable content such as harmful texts, personal information, and copyrighted material. To address this, machine unlearning aims to remove information from trained models. Recent work has shown that soft token attacks (STA) can successfully extract unlearned information from LLMs, but in this work we show that STAs can be an inadequate tool for auditing unlearning. Using common benchmarks such as Who Is Harry Potter? and TOFU, we demonstrate that in a strong auditor setting such attacks can elicit any information from the LLM, regardless of the deployed unlearning algorithm or whether the queried content was originally present in the training corpus. We further show that STA with just a few soft tokens (1-10) can elicit random strings over 400 characters long, indicating that STAs must be used carefully to effectively audit unlearning. Example code can be found at: https://github.com/IntelLabs/LLMart/tree/main/examples/unlearning
BibTeX
@inproceedings{chen2025soft,
  author = {Haokun Chen and Sebastian Szyller and Weilin Xu and Nageen Himayat},
  title = {{Soft Token Attacks Cannot Reliably Audit Unlearning in Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure