Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification

Vishnu Kabir Chhabra, Ding Zhu, Mohammad Mahdi Khalili

Abstract

Previous research has shown that fine-tuning language models on general tasks enhance their underlying mechanisms. However, the impact of fine-tuning on poisoned data and the resulting changes in these mechanisms are poorly understood. This study investigates the changes in a model's mechanisms during toxic fine-tuning and identifies the primary corruption mechanisms. We also analyze the changes after retraining a corrupted model on the original dataset and observe neuroplasticity behaviors, where the model relearns original mechanisms after fine-tuning the corrupted model. Our findings indicate that: (i) Underlying mechanisms are amplified across task-specific fine-tuning which can be generalized to longer epochs, (ii) Model corruption via toxic fine-tuning is localized to specific circuit components, (iii) Models exhibit neuroplasticity when retraining corrupted models on clean dataset, reforming the original model mechanisms.

Introduction

While fine-tuning is a standard practice to improve Large Language Model (LLM) performance, the internal "rewiring" that occurs during this process remains largely a black box. Most mechanistic interpretability research focuses on static, pre-trained models. This study, conducted by researchers at The Ohio State University, explores the dynamic evolution of internal circuits during both helpful and harmful fine-tuning. Specifically, it addresses how circuits evolve over epochs, the nature of internal corruption from "poisoned" data, and whether models can "heal" through a process akin to biological neuroplasticity.

Methodology

The study utilizes GPT-2-small as a subject, focusing on the Indirect Object Identification (IOI) task. In a sentence like "When Mark and Rebecca went to the garden, Mark gave flowers to...", the model must identify "Rebecca" as the indirect object.

The IOI Circuit

To understand the model's internal algorithm, the researchers mapped the IOI circuit, which consists of specialized attention heads such as Duplicate Token Heads, Induction Heads, S-Inhibition Heads, and Name Mover Heads.

The Indirect Object Identification (IOI) circuit in GPT-2 small, illustrating the flow of information between specialized attention heads.

The researchers employed techniques such as Path Patching, Activation Patching, and a new method called Cross-Model Activation Patching (CMAP) to isolate and observe the amplification of these specific mechanisms.

Datasets and Corruption

Experiments were conducted using a clean IOI dataset and two corrupted variants designed to break the model's logic: Name Moving, where the target is replaced with a random name, and Subject Duplication, where the target is replaced with the subject name (e.g., "...Mark gave flowers to Mark").

Key Insights: Amplification and Corruption

Circuit Amplification

Fine-tuning on clean data leads to Circuit Amplification. Rather than just learning new facts, the model recruits more attention heads and makes existing ones more specialized. For instance, "Backup Name Mover Heads"—which typically only activate if primary heads are damaged—begin to function as primary heads after fine-tuning.

Logit attribution and knockout effects comparing the original model to the amplified (fine-tuned) model.

Localized Corruption

When exposed to toxic data (poisoning), the model does not suffer a global failure. Instead, corruption is highly localized to specific components, such as the Name Mover Heads. The rest of the architecture remains intact, explaining why poisoned models can still appear functional in unrelated tasks.

Neuroplasticity and Recovery

A significant finding is the model's neuroplasticity. When a corrupted model is retrained on clean data, it doesn't merely find a new heuristic; it literally reforms the original circuits. These "relearned" circuits achieved 95-96% faithfulness to the original model's logic. This suggests that pre-training creates a strong structural "inertia" that allows models to revert to their original programming once clean data is reintroduced.

Conclusion

This research demonstrates that fine-tuning fundamentally alters model mechanisms through amplification and localized corruption. The discovery of neuroplasticity suggests that models possess an inherent structural resilience. While limited to GPT-2-small, these findings provide a foundation for developing better defense strategies against model poisoning and understanding the fundamental nature of transformer adaptation.

Summary updated

Abstract

Previous research has shown that fine-tuning language models on general tasks enhance their underlying mechanisms. However, the impact of fine-tuning on poisoned data and the resulting changes in these mechanisms are poorly understood. This study investigates the changes in a model's mechanisms during toxic fine-tuning and identifies the primary corruption mechanisms. We also analyze the changes after retraining a corrupted model on the original dataset and observe neuroplasticity behaviors, where the model relearns original mechanisms after fine-tuning the corrupted model. Our findings indicate that: (i) Underlying mechanisms are amplified across task-specific fine-tuning which can be generalized to longer epochs, (ii) Model corruption via toxic fine-tuning is localized to specific circuit components, (iii) Models exhibit neuroplasticity when retraining corrupted models on clean dataset, reforming the original model mechanisms.
BibTeX
@inproceedings{chhabra2025neuroplasticity,
  author = {Vishnu Kabir Chhabra and Ding Zhu and Mohammad Mahdi Khalili},
  title = {{Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=sbMzFaYqE4},
}
Figure