Abstract
PISCES (Precise In-parameter Suppression for Concept EraSure) is a surgical framework for erasing specific concepts directly from the internal parameters of Large Language Models (LLMs). By leveraging Sparse Autoencoders (SAEs) to disentangle complex model weights into interpretable, monosemantic features, PISCES removes unwanted knowledge with unprecedented precision. This approach leaves unrelated capabilities intact and makes the erasure significantly more robust against recovery through fine-tuning compared to existing methods.
Introduction
As Large Language Models (LLMs) scale, they inevitably ingest "toxic," sensitive, or copyrighted data during pretraining. Removing this information—a process known as concept erasure or unlearning—is critical for safety and compliance. Traditional methods like fine-tuning (e.g., RMU) or fact-level editing often act as "blunt instruments." They can be too coarse, accidentally damaging general reasoning or related knowledge (low specificity). Furthermore, these methods are often "shallow," meaning the knowledge remains latent and can be recovered via adversarial prompting or minimal additional training. The challenge lies in permanently and precisely deleting a concept without compromising the model's overall utility.
Methods
PISCES addresses the problem of "entanglement" in LLM Multi-Layer Perceptron (MLP) layers, where a single neuron often represents multiple disparate concepts. The framework follows a four-step process:
- Disentanglement: Using Sparse Autoencoders (SAEs), PISCES decomposes MLP weight vectors into thousands of fine-grained, "monosemantic" features.
- Identification: To isolate features belonging to a target concept (e.g., "Harry Potter"), the framework employs Vocabulary Projection (VocabProj). This identifies features that, when projected into the model's vocabulary space, strongly trigger tokens associated with the concept.
- Surgical Ablation: Once identified, these specific features are suppressed within the MLP parameters by setting their activations to negative values.
- Reconstruction: The modified features are used to reconstruct the MLP weights, which are then swapped back into the model.
Unlike "activation steering," which requires auxiliary models during inference, PISCES permanently modifies the model's weights.
Evaluation
The framework was evaluated on Gemma 2 (2B) and Llama 3.1 (8B) across 11 concepts, including pop culture (Harry Potter), sensitive topics (Guns, Pornography), and general knowledge (Culture of Greece). Evaluation metrics included MMLU for general knowledge, AlpacaEval for instruction following, and custom QA pairs for erasure efficacy.
Results
- High Efficacy and Specificity: PISCES reduced accuracy on target concepts to as low as 7.7% on Llama 3.1. It outperformed leading methods like RMU and ELM in specificity by up to 31%, preserving unrelated knowledge (e.g., Lord of the Rings knowledge remains intact after erasing Harry Potter).
- Robustness to Relearning: PISCES proved 28–38% more robust than baselines in "relearning" tests. While other models quickly "remembered" suppressed facts when fine-tuned, PISCES-edited models struggled to recover the data, suggesting the underlying "circuitry" was effectively removed.
- Efficiency: Due to the one-time identification step (VocabProj), PISCES is 1–2 orders of magnitude faster than fine-tuning when erasing multiple concepts simultaneously.
Discussion and Conclusion
PISCES demonstrates that SAEs are not just for interpretability but are powerful tools for precise model editing. By moving to the feature space, PISCES can delete specific components of a neuron's representation without affecting unrelated functions.
Current limitations include a focus on MLP layers, dependency on SAE reconstruction quality, and a brief manual filtering step. Future work aims to automate feature selection and extend the method to attention parameters. This research paves the way for more reliable and surgical safety interventions in large-scale neural networks.
Summary updated