Mitigating Copy Bias in In-Context Learning through Neuron Pruning

Ameen Ali, Lior Wolf, Ivan Titov

TL;DR

This paper introduces a novel method to combat "copying bias" in Large Language Models (LLMs) during few-shot in-context learning (ICL). By identifying and pruning specific "copying neurons" using Integrated Gradients on a synthetic task, the authors demonstrate consistent performance improvements across various LLM architectures and ICL tasks, suggesting a more generalizable and less error-prone learning process.

What Problem Does This Paper Address?

Large Language Models (LLMs) excel at few-shot in-context learning (ICL), meaning they can perform new tasks with only a few examples provided in the prompt. However, a significant issue arises: LLMs sometimes exhibit a "copying bias." Instead of learning the underlying pattern from the examples, they tend to directly copy the answers from the provided context. This is particularly problematic when the examples themselves might contain errors or when the task requires true generalization. This copying behavior hinders the model's ability to learn robustly and can lead to incorrect predictions, especially in scenarios with limited training data. The paper aims to address this by identifying and mitigating the neural mechanisms responsible for this shortcutting behavior.

Methods

The core of the proposed method involves two main steps:

  1. Identifying "Copying Neurons":
    • Synthetic Task: The authors create a synthetic task (vowel counting) that is designed to elicit copying behavior in LLMs. This task is chosen because it requires reasoning about structural attributes and is known to sometimes lead to copying errors.
  2. Integrated Gradients (IG): They employ the Integrated Gradients (IG) attribution method. IG is a standard technique used to understand which parts of the input or model contribute most to a specific output. Here, IG is used to trace the "copying errors" (when the model predicts an answer that is present in the prompt's examples but is incorrect) back to individual neurons.
  3. Prediction Shift: Specifically, IG is applied to the "prediction shift" – the difference between the probability of predicting an answer from the prompt's examples and the probability of predicting the true ground-truth answer. Neurons that significantly drive this shift are identified as "copying neurons."
  4. Mitigating Copying Bias via Neuron Pruning:
    • Pruning: Once identified, these "copying neurons" are pruned by setting their weights to zero. This is done without requiring task-specific data for neuron selection, making the method broadly applicable.
  5. Validation: A validation dataset is used to tune the optimal pruning rate and the specific layers (blocks) to prune, aiming to maximize validation accuracy.

The method is designed to be architecture-agnostic, working with both Transformer and State-Space Models without requiring architectural modifications.

Datasets

  • Synthetic Dataset: A synthetic dataset based on vowel counting is generated and used for identifying copying neurons. This dataset is crucial as it allows for neuron identification without relying on specific downstream task data, promoting generality.
  • Evaluation Datasets: The method is evaluated on a diverse set of 18 ICL tasks categorized into algorithmic, translation, linguistic, and knowledge tasks. These include tasks from Hendel et al. (2023) and Grazzi et al. (2024a), as well as real-world datasets like sentiment classification (SST2, SST5) and the object counting sub-task from the BBH benchmark.

Key Insights and Ideas

  • Task-Agnostic Copying Neurons: The research suggests that a small subset of neurons consistently prioritizes copying over generalization across various tasks and model architectures. This implies a shared, underlying mechanism for this shortcutting behavior.
  • Pruning Enhances Task Vectors: By pruning these copying neurons, the quality of "task vectors" (representations that capture task-specific information) is improved. This indicates that the pruned neurons were actively hindering the model's ability to form accurate task representations.
  • "Out-of-the-Box" Improvement: The proposed pruning method offers immediate performance gains without the need for task-specific fine-tuning or prompt engineering, making it a practical post-processing step.
  • Generalizability: The method's effectiveness across diverse LLM architectures (Transformers and State-Space Models) and a wide range of ICL tasks highlights its broad applicability.

Key Results

  • Consistent Performance Gains: Pruning the identified copying neurons consistently improved accuracy across a diverse set of ICL tasks and LLM architectures, including GPT2, Bloom, OPT, and Mamba models.
  • Reduced Copying Errors: The paper shows a significant reduction in copying errors after pruning, which in turn leads to a lower overall error rate. For instance, in Figure 2, the pruned models show a marked decrease in the shaded portion of the bars, representing copying errors.
  • Improved Task Vectors: Figure 5 demonstrates that pruning copying neurons leads to better task vector accuracies, suggesting that the pruning process helps the model better distill task-relevant information.
  • Minimal Performance Drop: The authors note that their pruning technique rarely leads to a performance drop, and when it does, the impact is marginal, making it a safe and effective intervention.

Limitations and Future Work

  • Limitations:
    • The method relies on a synthetic task for neuron identification. While this promotes generality, it might not perfectly capture all nuances of copying behavior across every possible real-world task.
  • The paper focuses on specific linear layers within model blocks. While effective, exploring other types of layers or more complex pruning strategies could yield further improvements.
  • Future Work:
    • The authors suggest further exploring the relationship between neuron-level representations and higher-level task embeddings (task vectors) to understand how pruning influences this dynamic.
  • Investigating how this pruning strategy can be integrated into LLM deployment pipelines as a standard post-processing step is a promising direction.
  • Exploring the potential for disentangling multiple activation pathways through targeted interventions could lead to even more sophisticated reasoning capabilities.

Summary updated

Abstract

Large language models (LLMs) have demonstrated impressive few-shot in-context learning (ICL) abilities. Still, we show that they are sometimes prone to a `copying bias', where they copy answers from provided examples instead of learning the underlying patterns. In this work, we propose a novel and simple method to mitigate such copying bias. First, we create a synthetic task and use the Integrated Gradients method to identify neurons that prioritize copying over generalization. We demonstrate that pruning these neurons consistently improves performance across a diverse set of ICL tasks. We also show that our method is applicable across various LLM architectures, including Transformers and State-Space Models, without requiring modifications. In our analysis, we adopt a task-recognition perspective on ICL and examine task vectors (Hendel et al., 2023) induced by the model. We find that pruning enhances the quality of these vectors, suggesting that the pruned neurons previously hindered effective task recognition.
BibTeX
@inproceedings{ali2026mitigating,
  author = {Ameen Ali and Lior Wolf and Ivan Titov},
  title = {{Mitigating Copy Bias in In-Context Learning through Neuron Pruning}},
  booktitle = {EACL (Findings)},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.13},
}
Figure