Breaking Down Bias: On The Limits of Generalizable Pruning Strategies

Sibo Ma, Alejandro Salinas, Julian Nyarko, Peter Henderson

Abstract

As Large Language Models (LLMs) are increasingly integrated into critical sectors like finance and healthcare, their tendency to mirror societal biases poses a significant risk. This research investigates whether model pruning—the selective deactivation of specific neurons or attention heads—can effectively eliminate racial bias. While the study finds that "surgical" pruning can nearly eliminate bias within specific, narrow contexts, the effectiveness of these interventions decays significantly when applied across different scenarios. These findings suggest that bias is encoded in a highly context-specific manner, challenging the feasibility of a "one-size-fits-all" mitigation strategy.

Introduction

When prompted with names associated with different racial groups, LLMs often produce disparate outcomes, such as offering lower price estimates or different financial advice based on perceived race. A central technical question arises: Is bias encoded in a specific set of "biased neurons" that can be removed, or is it a diffuse property of the model? This paper tests the limits of generalizable pruning to determine if a model can be "fixed" once for all tasks or if mitigation must be tailored to every specific use case.

Methodology: The Surgical Approach

The researchers employed a three-step approach to bias mitigation using the Llama-3-8B-Instruct model. The process focuses on identifying and removing components that drive discriminatory behavior without compromising the model's general utility.

  1. Scoring: Using the WandA (Weights and Activations) method, the researchers identified which neurons and attention heads were most influential when the model processed prompts containing names associated with specific racial groups (Black vs. White).
  2. Localization: They isolated "bias-influential" components by identifying neurons that were highly active for minority-associated prompts but inactive for majority-associated prompts.
  3. Pruning: These components were "zeroed out." The researchers compared fine-grained neuron-based pruning against coarse-grained attention head-based pruning.
Figure 1: The pruning workflow for bias mitigation. By identifying and removing components exclusive to minority group activations, the model's price distribution for different racial groups is equalized.

The Generalization Gap

The study utilized a dataset of 64 racially-associated names across four distinct scenarios: Purchase (negotiating prices), Activities (budgeting time), Services (estimating costs), and Finance (determining savings rates).

A critical discovery was the "Generalization Gap." While a pruning mask developed for a specific "car purchase" scenario could reduce bias to near zero in that context, its effectiveness dropped by approximately 60% when applied to other commercial transactions. This indicates that the model does not have a single "bias center"; instead, it likely possesses distinct sets of neurons for "medical-bias," "finance-bias," and "hobby-bias" that only partially overlap.

Key Results and Insights

  • Granularity Matters: Neuron-level pruning was significantly more effective than pruning entire attention heads. This suggests that bias is encoded at a very granular level within the model's Multi-Layer Perceptron (MLP) layers, specifically in the mid-to-late layers of the network.
  • In-Context Success vs. Cross-Context Decay: In-context pruning reduced the Standardized Mean Difference (SMD) of bias from -0.59 to +0.07 (near parity). However, when the same mask was applied across different contexts, the SMD only improved to -0.37.
  • Utility Preservation: The model's general intelligence remained intact after the "surgery," with an "Inlier Ratio" (a measure of output coherence) remaining above 0.98.

Conclusion and Policy Implications

The contextual nature of AI bias has significant implications for both technical development and policy. Because a developer cannot prune a base model for every possible downstream scenario, the researchers argue that the legal and ethical responsibility for AI fairness may need to shift toward the deployer—the entity using the AI for a specific, localized task.

Future research may look toward Sparse Autoencoders (SAEs) to find more interpretable and potentially more generalizable features of bias, as well as expanding the scope beyond binary racial comparisons to include intersectional identities and non-English contexts.

Summary updated

BibTeX
@inproceedings{ma2025breaking,
  author = {Sibo Ma and Alejandro Salinas and Julian Nyarko and Peter Henderson},
  title = {{Breaking Down Bias: On The Limits of Generalizable Pruning Strategies}},
  booktitle = {FAccT},
  year = {2025},
  doi = {10.1145/3715275.3732161},
}
Figure