Abstract
Large Language Models (LLMs) often inadvertently memorize sensitive or toxic information during training. While Gradient Ascent (GA) is a common technique for unlearning this data, it often results in "over-unlearning," degrading the model's general linguistic capabilities. We present Fine-grained Pluggable Gradient Ascent (FPGA), a surgical unlearning method that employs token-level weighting and external discriminators. FPGA targets only sensitive components of a sequence, effectively removing undesirable knowledge while preserving model utility and reasoning. Our experiments demonstrate that FPGA achieves superior forgetting metrics and maintains higher performance on downstream tasks compared to standard GA.
Introduction: The Challenge of Knowledge Unlearning
As LLMs are trained on massive datasets scraped from the internet, they inevitably ingest sensitive information, including Personally Identifiable Information (PII) and toxic content. Removing this data without the prohibitive cost of retraining from scratch is the goal of "knowledge unlearning."
The current standard method, Gradient Ascent (GA), operates by maximizing the loss on a target sequence. However, GA is an indiscriminate "sledgehammer." It treats every token in a sentence—including neutral filler words—as equally problematic. This leads to over-unlearning, where the model's underlying linguistic foundations are corrupted, causing a significant drop in general reasoning and text generation quality.
FPGA: A Surgical Approach to Unlearning
To address the limitations of GA, we introduce Fine-grained Pluggable Gradient Ascent (FPGA). Instead of treating all tokens as equal targets for deletion, FPGA assigns adaptive weights to each token based on its sensitivity. This "scalpel" approach allows the model to forget specific facts (e.g., a specific address) while retaining the ability to structure sentences about addresses in general.
Methodology
The FPGA framework consists of three primary components designed to refine the unlearning process:
- Selective Masking: The model identifies the top-$m$ most likely next tokens. This ensures the unlearning process focuses on the tokens the model is most "confident" about, which are typically the ones it has memorized most strongly.
- Discriminator Evaluation: This is the core innovation. An external, frozen discriminator (such as a toxicity detector or PII scanner) evaluates the tokens. Tokens that trigger the discriminator receive higher weights in the loss function, while neutral tokens are largely ignored.
- Pluggable Integration: FPGA is designed as a modular loss function. It can be used as a standalone objective or combined with regularization techniques (FPGA-R), such as KL-divergence, to anchor the model to its original state and prevent performance drift.
Experimental Evaluation
We evaluated FPGA across two main dimensions: unlearning efficacy and the preservation of general model utility.
Datasets and Benchmarks
- Unlearning Targets: Specialized versions of GPT2-small and GPT-NEO were fine-tuned on toxic data and PII.
- Utility Benchmarks: To ensure general intelligence was preserved, we tested on 9 classification datasets (including Hellaswag and ARC) and 4 dialogue tasks (including Wizard of Wikipedia).
- Discriminator Training: A toxicity discriminator was trained using the Jigsaw Unintended Bias dataset.
Key Results
Our results indicate that FPGA outperforms standard GA in both forgetting and utility:
- Superior Forgetting: In toxicity unlearning on GPT2-small, FPGA achieved a Memorization Accuracy (MA) of 14.03%, outperforming GA's 15.83%.
- Utility Preservation: While GA caused dialogue performance to plummet (F1 of 8.46), FPGA maintained a higher F1 of 8.53. With regularization (FPGA-R), performance in some benchmarks even exceeded the original model.
- Behavior Alignment: FPGA proved capable of performing broad behavior alignment (reducing overall toxicity) similar to RLHF, but without the need for expensive preference-labeled data.
Conclusion and Future Work
FPGA offers a precise and efficient alternative to standard unlearning methods. By targeting sensitive tokens rather than entire sequences, it mitigates the "over-unlearning" problem that plagues current techniques. Future research will explore integrating FPGA with Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA to further reduce the computational overhead for end-users.
Summary updated