Large Language Models Relearn Removed Concepts

Michelle Lo, Fazl Barez, Shay B. Cohen

Introduction

As Large Language Models (LLMs) are increasingly integrated into sensitive applications, the ability to "edit" these models—specifically to remove undesirable or unsafe concepts—has become a priority for AI safety researchers. A prevalent technique for concept removal is neuron pruning, which involves identifying and neutralizing specific neurons responsible for a given concept. However, a critical question remains: is this removal permanent? This paper investigates the phenomenon of neuroplasticity in LLMs, exploring whether models can recover lost information when subjected to further training after pruning.

Methodology

The researchers developed a systematic framework to induce and monitor neuroplasticity through four distinct phases:

  1. Identification: Utilizing the NeuroX toolkit, the authors performed a "probeless" corpus-based search to rank neurons based on their correlation with specific concepts, such as geographical locations.
  2. Ablation: The most salient neurons identified in the previous step were pruned by setting their weights to zero, effectively "damaging" the model's ability to process those concepts.
  3. Retraining: The pruned model underwent retraining on its original task to observe if and how it recovered its lost capabilities.
  4. Analysis: The recovery process was tracked using two primary metrics: Concept Saliency (the degree to which a neuron encodes a concept relative to the model) and Concept Similarity (using Word2Vec to compare the semantic profiles of neurons before and after retraining).

The study utilized the CoNLL-2003 dataset for Named Entity Recognition (NER) tasks and BERT ConceptNet to provide ground-truth labels for latent concepts learned by the models.

Key Insights into Model Recovery

The study reveals that LLMs possess a startling capacity for recovery, characterized by several key mechanisms:

Rapid Performance Recovery

The "rehabilitation" of a model does not require extensive retraining. In experiments with DistilBERT, pruning location-specific neurons caused the F1 score for location recognition to drop from 0.937 to a negligible 0.034. However, after only two epochs of retraining, the performance surged back to 0.915, nearly reaching the original baseline. This suggests that the underlying capacity to process the concept is not destroyed, but merely displaced.

The Upstream Shift

In unedited models, complex conceptual representations typically reside in the later layers. The research found that after pruning, the model relocates these representations to earlier layers. This "upstream shift" suggests that the model compensates for missing specialized "hardware" at the end of the pipeline by processing the concept sooner in the network. For instance, in DistilBERT, the primary hub for location concepts shifted from Layer 6 to Layer 4 post-retraining.

Semantic Priming and Polysemanticity

Neurons do not take over lost concepts at random. The neurons that emerge as new "concept hubs" are typically those that were already capturing semantically related information—a process termed semantic priming. Most neurons that reacquired the pruned concept showed similarity scores > 0.5 compared to their original state, confirming they were already "primed" for the task. Furthermore, these neurons often become polysemantic, blending their original functions with the newly acquired concept. A neuron that previously responded to the token "Egyptian" might adapt to respond to both "Egyptian" and "Damascus" after the model's location neurons are pruned.

Discussion and Limitations

While the findings demonstrate the resilience of conceptual representations in LLMs, the authors note several limitations. The study focused on relatively small architectures, such as DistilBERT and GPT-2; it remains to be seen if massive models like GPT-4 exhibit similar patterns of rapid redistribution. Additionally, the experiments were centered on Named Entity Recognition, and neuroplasticity may manifest differently in abstract reasoning or purely generative tasks. Finally, while there is a strong correlation between "primed" neurons and relearning, the exact causal mechanism of why specific neurons are chosen for remapping requires deeper investigation.

Conclusion

The phenomenon of neuroplasticity suggests that simple neuron pruning may only provide a temporary mask rather than a permanent removal of concepts. For AI safety measures to be effective, researchers must develop techniques that "lock" model edits, preventing the model from repurposing existing neurons to circumvent safety-oriented interventions during subsequent fine-tuning.

Summary updated

Abstract

Advances in model editing through neuron pruning hold promise for removing undesirable concepts from large language models. However, it remains unclear whether models have the capacity to reacquire pruned concepts after editing. To investigate this, we evaluate concept relearning in models by tracking concept saliency and similarity in pruned neurons during retraining for named entity recognition tasks. Our findings reveal that models can quickly regain performance post-pruning by relocating advanced concepts to earlier layers and reallocating pruned concepts to primed neurons with similar semantics. This suggests that models exhibit polysemantic capacities and can blend old and new concepts in individual neurons. While neuron pruning provides interpretability into model concepts, our results highlight the challenges of permanent concept removal for improved model *safety*. Monitoring concept reemergence and developing techniques to mitigate relearning of unsafe concepts will be important directions for more robust model editing. Overall, our work strongly demonstrates the resilience and fluidity of concept representations in LLMs post concept removal.
BibTeX
@inproceedings{lo2024large,
  author = {Michelle Lo and Fazl Barez and Shay B. Cohen},
  title = {{Large Language Models Relearn Removed Concepts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.492},
}
Figure