Toward Inclusive Language Models: Sparsity-Driven Calibration for Systematic and Interpretable Mitigation of Social Biases in LLMs

Prommy Sultana Hossain, Chahat Raj, Ziwei Zhu, Jessica Lin, Emanuela MarascoEMNLP 2025

TL;DR

This paper introduces a novel, architecture-aware method to identify and mitigate social biases in Large Language Models (LLMs) by focusing on specific attention heads within the transformer architecture. By precisely calibrating these "bias-amplifying" components, the authors achieve significant bias reduction without compromising model performance, offering a more interpretable and efficient approach than existing methods.

What Problem Does This Paper Address?

Large Language Models (LLMs) have become incredibly powerful, but they often inherit and amplify harmful societal biases present in their training data. These biases can manifest as stereotypes related to gender, race, age, and other protected attributes, leading to unfair or discriminatory outputs. While various debiasing methods exist, they often struggle with scalability, compromise general model performance, or lack interpretability. This paper aims to address these limitations by proposing a systematic, component-level approach to identify and mitigate these biases in a targeted and understandable manner.

Methods

The authors propose a four-step framework:

  1. Context Influence (CI) Score: They introduce a novel metric to quantify how contextual cues influence model predictions, revealing bias patterns across different demographic contexts.
  2. Correlation-Based Analysis: They correlate the CI scores with the model's internal activations, specifically focusing on attention heads, to pinpoint those most responsible for propagating biased associations.
  3. Temporal Bias Localization: This step visualizes how identified problematic attention heads process and amplify bias across the model's architecture, providing architectural insights.
  4. Targeted Calibration (Soft Pruning): They implement a "soft pruning" strategy that selectively scales the parameters of implicated attention heads. This is followed by lightweight fine-tuning to maintain fluent text generation. This approach avoids retraining from scratch and does not require access to proprietary model internals.

Datasets

The paper utilizes several datasets for training and evaluation:

  • BBQ (Bias Benchmark for Question Answering): Used for training and evaluating bias, it contains ambiguous and disambiguated contexts across nine demographic categories.
  • StereoSet: Used for evaluating stereotype bias through associations between contexts and stereotypical/anti-stereotypical sentences, covering four domains.
  • CrowS-Pairs: Used for validation, it consists of minimally edited sentence pairs differing in demographic references.
  • MMLU (Massive Multitask Language Understanding): Used to assess general knowledge and reasoning capabilities.
  • HellaSwag: Used to measure commonsense reasoning through sentence completion tasks.

Key Insights and Ideas

  • Attention Heads as Bias Pathways: The research strongly suggests that specific attention heads, rather than the entire model or hidden states, are the primary conduits for bias propagation.
  • Architecture-Awareness is Key: Understanding the distinct roles of different architectural components (like attention heads) is crucial for effective and targeted bias mitigation.
  • Sparsity-Driven Calibration Works: Selectively "pruning" or scaling parameters in identified bias-amplifying attention heads is an effective way to reduce bias without significant performance degradation.
  • Bias is Not Uniform: Even larger models exhibit significant biases across demographic subgroups, indicating that simply increasing parameter count does not inherently solve the problem.
  • Interpretability Matters: The method's focus on identifying specific attention heads and visualizing their behavior makes the debiasing process more interpretable and trustworthy.

Key Results

The proposed method demonstrated significant improvements across multiple LLMs:

  • Bias Reduction: Up to 37% on BBQ, 32% on StereoSet, and 33% on CrowS-Pairs.
  • Performance Preservation/Improvement: Simultaneously improved reasoning performance on MMLU by up to 10%.
  • Superiority over Baselines: Consistently achieved lower bias scores than established methods like INLP, Layer-wise Debiasing, and Self-Debiasing across various models and datasets.
  • Effectiveness Across Models: The approach showed consistent superiority across five pre-trained models, including LLaMA, Aya, and Qwen.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Layer-Agnostic Pruning: The current pruning strategy aggregates behavior across all layers. Future work could explore layer-specific pruning thresholds to better account for the distinct functional roles of early vs. late layers, potentially preserving more desirable functionality.
  • Global vs. Category-Specific Calibration: The pruning is based on global correlation with CI scores. Future work could explore category-specific calibration to address subtler or contextually entangled biases in specific dimensions.
  • Exclusive Focus on Attention Mechanisms: While attention heads are identified as primary bias pathways, other components like MLPs and hidden states are not directly intervened upon. Future research could explore multi-component calibration, though this may be more challenging due to entanglement and interpretability issues.
  • Limited Fine-Tuning Scope: The final fine-tuning stage uses a small set of unbiased examples. While effective, future work could explore continual learning frameworks for more robust debiasing consolidation.
  • Lack of Category-Specific Feedback Loop: The current pipeline lacks real-time feedback on bias reduction effectiveness across individual demographic categories during mitigation. Incorporating such a loop could allow for dynamic adjustment of pruning strategies.

Summary updated

BibTeX
@inproceedings{hossain2025toward,
  author = {Prommy Sultana Hossain and Chahat Raj and Ziwei Zhu and Jessica Lin and Emanuela Marasco},
  title = {{Toward Inclusive Language Models: Sparsity-Driven Calibration for Systematic and Interpretable Mitigation of Social Biases in LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure