LLM Bias Detection and Mitigation through the Lens of Desired Distributions

Ingroj Shrestha, Padmini SrinivasanEMNLP 2025

TL;DR

This paper proposes a novel approach to detect and mitigate bias in Large Language Models (LLMs) by focusing on aligning their output distributions with desired distributions, rather than solely aiming for equal representation. They introduce a weighted adaptive KL divergence loss function that effectively reduces gender-profession bias while preserving language modeling capabilities, showing significant improvements over existing methods.

What Problem Does This Paper Address?

The prevailing approach to bias mitigation in LLMs often centers on achieving social equality, such as demographic parity or equalized odds. However, this overlooks scenarios where aligning with real-world distributions is crucial for factual grounding and model reliability. For instance, in healthcare, gender might be a relevant factor, making a 50-50 split undesirable. The authors highlight this gap, arguing that bias should be understood as any deviation from a desired distribution, which can be either an equal distribution or a real-world distribution, depending on the application. They aim to address this by developing methods to fine-tune LLMs to match these specific target distributions.

Methods

The core of the paper lies in their proposed bias mitigation strategy: a weighted adaptive KL divergence loss function for fine-tuning LLMs.

  • Bias Detection: They quantify bias using the Kullback-Leibler (KL) divergence between the LLM's predicted gender-profession distribution and a desired distribution (either equal or real-world). This is calculated for each profession and then averaged to get a final bias score.
  • Bias Mitigation (Weighted Adaptive KL Loss):
    • Non-adaptive KL Loss: A baseline approach that uses KL divergence to guide the model towards the desired distribution, giving equal weight to all professions.
  • Weighted Adaptive KL Loss: This is their novel contribution. It dynamically adjusts the loss based on group-specific dynamics. It incorporates:
  • Adaptive Loss: The loss for a profession category is adjusted based on its historical KL divergence, preventing groups with consistently high divergence from dominating the overall loss.
  • Adaptive Loss Scaling: A small constant is added to the denominator to control update magnitudes, allowing for larger or smaller, more cautious updates based on the group's KL divergence.
  • Stability-Aware Weighting: A weighting factor is computed based on the variance of KL divergence, assigning lower weights to less stable predictions (higher variance) and higher weights to more stable ones.
  • MLM Loss (for MLMs): A secondary objective is added to preserve language modeling capability by incorporating Masked Language Model (MLM) loss. This is not needed for Autoregressive Language Models (ALMs) as their inherent objective already preserves this capability.

Datasets

  • Profession Data: Derived from the U.S. Bureau of Labor Statistics (2024), comprising 225 professions. These were grouped into:
  • Male-dominated (DPmale: [0-30%])
    • Female-dominated (DPfemale: [70,100%])
    • Balanced (DPbalanced: [45,55%])
  • Templates: Six templates were used for bias analysis and fine-tuning, with a split between common and rare templates for training/validation and testing.
  • External Corpora for Evaluation:
    • GAP Corpus
    • WikiText-103 (development and test sets)
  • Downstream Tasks for Evaluation:
    • MLMs: GLUE tasks
    • ALMs: LM Evaluation Harness (covering HellaSwag, LAMBADA, TruthfulQA, MMLU, and GLUE)

Key Insights and Ideas

  • Redefining Bias: The paper compellingly argues that bias isn't just about achieving equality, but about deviating from a desired distribution, which can be real-world. This broadens the scope of bias mitigation significantly.
  • Real-World Alignment is Crucial: For applications where factual accuracy and domain-specific realities are paramount (e.g., healthcare), aligning with real-world distributions is more important than enforcing strict equality.
  • Adaptive Loss for Balanced Mitigation: The weighted adaptive KL loss is a key innovation. It intelligently balances the mitigation effort across different profession groups, preventing over-correction in some and under-correction in others, leading to more robust and equitable debiasing.
  • Preserving Language Modeling: A critical aspect is maintaining the LLM's core language modeling capabilities. The authors demonstrate that their methods achieve this, often with minimal degradation, and even show improvements in some cases when MLM loss is incorporated.
  • ALMs vs. MLMs: The paper highlights differences in debiasing strategies for MLMs and ALMs, particularly regarding the need for an explicit MLM loss to preserve language modeling in MLMs.

Key Results

  • Equal Distribution Mitigation: For the "equal distribution" target, their methods achieve near-complete bias removal (over 98% reduction) across MLMs with minimal degradation in language modeling performance.
  • Real-World Distribution Mitigation: For the "real-world distribution" target, they achieved significant bias reduction, ranging from 30-75% across different models and profession categories.
  • ALM Performance: Autoregressive LLMs (Llama3, Qwen2) showed minimal bias under the equal distribution but notable bias under the real-world distribution. Their fine-tuning approach achieved 50-62% reduction in bias for Llama Instruct models.
  • Language Modeling Preservation:
    • For MLMs, MLM loss increased by a small margin (2.3%-14.5%), and downstream GLUE scores remained largely consistent.
  • For ALMs, perplexity degradation was minimal (e.g., 1.3-3% for Llama3.1-8B-Instruct on WikiText-103-test), and downstream performance was maintained.
  • Ablation Studies: Removing adaptive loss scaling slightly degraded bias mitigation, while adding MLM loss for MLMs improved language modeling performance with minimal impact on bias mitigation.

Limitations and Future Work

  • Binary Gender Focus: The analysis is limited to binary gender due to data availability. Future work should extend to other demographic groups (e.g., race) and non-binary gender identities.
  • US-Centric Data: The reliance on US-based distributions might reinforce a geographically constrained perspective. Incorporating non-US contexts is necessary for international deployment.
  • Limited Professions and Templates: The analysis used 225 professions and six templates. Expanding these could yield broader insights.
  • Template-Based Approach: The current method relies on templates. Future research should explore bias mitigation for open-ended generation, which presents additional complexities in maintaining generation quality and contextual relevance.
  • Out-of-Distribution Scenarios: The study did not comprehensively evaluate out-of-distribution scenarios like template variation.
  • Fine-tuning MLMs: Using a larger external corpus for fine-tuning MLMs could better preserve language modeling capabilities.
  • ALM Exploration: Extending the bias mitigation to a wider range of ALMs is an important future direction.

Summary updated

BibTeX
@inproceedings{shrestha2025llm,
  author = {Ingroj Shrestha and Padmini Srinivasan},
  title = {{LLM Bias Detection and Mitigation through the Lens of Desired Distributions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure