Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?

Zhengyang Shan, Aaron Mueller

Introduction

As Large Language Models (LLMs) are increasingly integrated into high-stakes domains like recruitment and admissions, the persistence of harmful stereotypes—such as associating specific races or genders with certain professions—remains a critical challenge. Traditional debiasing approaches often face a significant trade-off: retraining is computationally expensive, while the blunt removal of demographic information can impair the model’s performance in legitimate contexts where demographic recognition is necessary (e.g., pronoun resolution in translation).

This research explores the concept of mechanistic independence. The central question is whether the internal neural circuitry responsible for demographic knowledge is distinct from the circuitry that applies stereotypical biases. If these pathways are independent, it should be possible to "surgically" ablate bias without erasing fundamental demographic understanding.

Methodology: Identifying Bias Features

The researchers employed Sparse Autoencoders (SAEs) to decompose the complex activations of LLMs into thousands of interpretable features. They investigated two primary methods for identifying "bias features":

  1. Attribution-based: Utilizing gradient-based methods (Integrated Gradients) to pinpoint features that causally drive the model toward a specific demographic prediction.
  2. Correlation-based: Identifying features that consistently activate in the presence of specific demographic mentions, regardless of the task.
The three-stage pipeline: (1) Feature Analysis to identify bias-related features via attribution or correlation, (2) Ablation of selected features, and (3) Evaluation of bias reduction and control task accuracy.

Following identification, the researchers performed intervention experiments by "ablating" (zeroing out) these features during inference. The goal was to reduce bias while maintaining accuracy on control tasks, such as simple name-to-gender recognition.

Experimental Setup and Datasets

The study utilized a multi-task evaluation framework:

  • Demographic Identifiers: A dataset of names associated with various races (including a new set of 100 Asian names) and genders.
  • Stereotypical Tasks: Professional associations using 41 professions from the WinoBias dataset and educational attainment levels based on U.S. Bureau of Labor Statistics data.
  • Validation: The Winogender benchmark was used to assess generalization on naturalistic pronoun-resolution tasks.

Key Results and Insights

Experiments conducted on Gemma-2-9B revealed several critical findings regarding the nature of internal model bias.

Surgical Debiasing Success

The attribution-based ablation method successfully reduced profession-based bias by 6% to 34%. Remarkably, this was often achieved without degrading—and sometimes even improving—name-recognition accuracy. For instance, gender-based debiasing saw a 6.1% reduction in bias alongside a 0.8% boost in control accuracy.

Comparison of bias reduction (Δ% KL) and control accuracy (Δ% Accuracy) for gender and race tasks across different feature selection methods.

Task-Specific Mechanisms

The study found that features driving stereotypes are often distinct from those used for basic demographic recognition. Qualitative analysis suggested that models frequently use "formal discourse markers" or "technical terminology" as proxies for demographics or competence. Ablating these specific pathways allows for debiasing without erasing the underlying demographic concepts.

The "Prior Collapse" Phenomenon

A notable exception occurred in education-based tasks. When causal features were removed, the model sometimes became more biased by defaulting to the most frequent label in its training data (e.g., "Bachelor's degree"). This "prior collapse" suggests that for certain types of bias, correlation-based identification is superior, achieving a 30.7% bias reduction where attribution failed.

Conclusion and Future Work

This work demonstrates that LLM bias is often mechanistically independent of core demographic knowledge, allowing for targeted interventions. However, the researchers noted several limitations:

  • Scope: The study focused on a limited set of categories (Race, Gender, Education) in English-language models.
  • SAE Coverage: Sparse Autoencoders may not capture the entirety of a model's internal state, potentially leaving some bias circuitry hidden.
  • Generalization: Future research is needed to automate the selection of the best debiasing strategy (attribution vs. correlation) for different types of bias and intersectional identities.

Summary updated

BibTeX
@inproceedings{shan2026measuring,
  author = {Zhengyang Shan and Aaron Mueller},
  title = {{Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?}},
  booktitle = {EACL (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.199},
}
Figure