Mitigating Copy Bias in In-Context Learning through Neuron Pruning
Abstract
Large Language Models (LLMs) often "cheat" during in-context learning (ICL) by copying answers directly from provided examples instead of learning the underlying rule—a phenomenon called "copy bias." This paper introduces a simple yet effective method to identify and prune a small subset of "copying neurons" using Integrated Gradients, significantly boosting the model's ability to generalize across diverse tasks and architectures.
Introduction: The Challenge of Copy Bias
In-Context Learning (ICL) allows LLMs to solve new tasks simply by looking at a few examples in a prompt. However, ICL is notoriously brittle. One major reason is copy bias: instead of figuring out the logic (e.g., "count the vowels in this word"), the model often defaults to a shortcut, picking one of the labels already present in the prompt.
While researchers have tried to fix this through better prompt engineering or calibration, we still don't fully understand the internal mechanics of why models take these shortcuts. This paper shifts the focus from the prompt to the model's internal architecture, investigating whether specific neurons are responsible for this lazy behavior.
Methodology: Identifying and Pruning Copying Neurons
The authors propose a three-step pipeline to detect and mitigate copy bias:
- The Proxy Task: They use a synthetic "vowel-counting" task to trigger copying behavior. Because this task requires structural reasoning that LLMs sometimes struggle with, it provides a clear signal when the model "cheats" by copying a previous answer.
- Neuron Identification: Using Integrated Gradients (IG)—an attribution method—they trace "prediction shifts." Specifically, they look for neurons that contribute most to the model shifting its prediction away from the correct answer and toward a label found in the prompt examples.
- Pruning: Once these "copying neurons" are identified, they are simply deactivated (their weights are set to zero). Crucially, the authors found that these neurons are task-agnostic.
This approach was tested across multiple architectures, including standard Transformers (GPT-2, BLOOM, OPT, LLaMA) and newer State-Space Models (Mamba).
Experimental Setup and Datasets
The study utilizes a wide array of benchmarks to prove generalizability:
- Synthetic ICL Tasks: 18 tasks across three categories: Algorithmic (e.g.,
to_lowercase), Linguistic (e.g.,singular_to_plural), and Knowledge-based (e.g.,country_to_capital). - Real-world Benchmarks: Sentiment classification datasets (SST2, SST5) and the "Object Counting" task from the Big-Bench Hard (BBH) benchmark.
- Proxy Dataset: A custom-built vowel-counting dataset used specifically for the initial neuron identification phase.
Key Results and Insights
Pruning a tiny percentage of neurons (1-10%) consistently improves performance, suggesting that copy bias is a specific mechanical pathway the model uses to minimize loss during training. The most surprising finding is that the same neurons responsible for copying in a simple vowel-counting task are the ones hindering the model in complex sentiment analysis.
Key performance highlights include:
- Consistent Accuracy Gains: Pruning led to improvements across almost all tested models. For example, on the "Letter to Uppercase" task, OPT-350M's 1-shot accuracy jumped from 0.24 to 0.51.
- Reduction in Copying Errors: In models like GPT2-Small, the "copying error rate" dropped significantly, leading to a lower total error rate overall.
- Architecture Agnostic: The method worked just as well on Mamba (SSMs) as it did on Transformers, suggesting that copy bias is a fundamental trait of how these models learn.
Conclusion and Future Work
By pruning copying neurons, the authors analyzed "task vectors" and found they became "cleaner," proving that these neurons were actively muddying the model's ability to recognize task logic. While the current method requires a validation set to find optimal pruning parameters, future work could explore "dynamic pruning," where neurons are only suppressed when the model detects a high likelihood of a copying shortcut.
Summary updated