Collapsed Language Models Promote Fairness

Jingxuan Xu, Wuyang Chen, Linyi Li, Yao Zhao, Yunchao Wei

TL;DR

Researchers have discovered that "fairer" language models naturally exhibit a geometric phenomenon called Neural Collapse, where representations of gender-sensitive words align more tightly with their corresponding embeddings. By explicitly enforcing this "collapse" through a simple, plug-and-play regularization term during fine-tuning, they significantly improved model fairness across multiple benchmarks without sacrificing general language processing performance.

The Challenge of Measuring and Enforcing Fairness

Pre-trained language models (PLMs) like BERT and RoBERTa are notorious for absorbing societal biases from their massive training corpora. These biases often link specific demographic groups to certain roles or traits (e.g., associating "doctor" with men and "nurse" with women).

While many debiasing techniques exist—ranging from data augmentation to specialized prompts—the field lacks a "principled" understanding of what a debiased model actually looks like under the hood. There is no shared geometric property that defines a fair model. This paper seeks to bridge that gap by applying Neural Collapse (NC) theory to the problem of fairness, asking if we can use the geometric structure of a model's internal representations to both understand and enforce fairness.

Methods: Leveraging Neural Collapse

The authors leverage the concept of Neural Collapse, a phenomenon originally observed in image classification where, at the end of training, the last-layer features of the same class "collapse" into a single point (the class mean), and these means align perfectly with the classifier's weights.

Visualizing Neural Collapse: Debiased representations (solid dots) align more tightly with classifier weights (arrows) compared to biased ones (hollow circles).

In language models, the authors treat next-token prediction as a massive classification task. They focus specifically on a metric called $(U)NC_3$ (Uniform Duality), which measures the alignment between:

  1. Class Means ($\mu$): The average representation of a specific token (like "he" or "she") across different contexts.
  2. Classifier Weights ($w$): The word embeddings in the model's final layer.

The Innovation: The authors found that existing debiased models already showed higher $(U)NC_3$ alignment for gendered words. They turned this observation into a method: a new regularization loss ($\mathcal{L}_{NC3}$) that explicitly minimizes the variance of this alignment for a predefined list of fairness-sensitive words. This method is task-agnostic and can be added to almost any existing fine-tuning pipeline as an auxiliary objective.

Key Insights and Ideas

  • Fairness has a Geometric Signature: Debiasing isn't just about changing output probabilities; it fundamentally alters the geometry of the embedding space. In fair models, the "gap" between how a model represents a word and how it classifies that word disappears for sensitive terms.
  • The "Classifier" is the Key: Standard Neural Collapse metrics often fail in language models because the data is too complex. However, the authors found that metrics involving the word embeddings (classifier weights) are remarkably stable and highly correlated with fairness.
  • Selective Collapse: Interestingly, debiased models don't collapse the entire vocabulary—only the words related to the bias being addressed (e.g., gendered pronouns and nouns). This suggests that fairness is encoded locally in the representation space.

Key Results

The proposed NC-based regularization consistently improved fairness across various baseline models (Mabel, ASE, and BEC) evaluated on datasets like SNLI, MNLI, OntoNotes 5.0, and TinyStories.

StereoSet evaluation results showing improvements in the ICAT score when applying the proposed NC regularization.
  • StereoSet: Adding NC regularization to the "Mabel" model improved its ICAT score (a metric balancing language modeling and fairness) from 73.07 to 74.55.
  • WinoBias: On this coreference task, the "Mabel + NC" model showed a massive improvement, reducing the bias gap (TPR-2) from 3.41 to 1.97.
  • Bias-in-Bios: The method consistently reduced the disparity in true positive rates between genders, achieving a state-of-the-art GAP score of 0.455 when combined with Mabel.
  • Performance Preservation: Crucially, scores on the GLUE benchmark remained nearly identical to the baselines, proving that the model can become fairer without becoming "dumber" at core linguistic tasks.

Limitations and Future Work

  • Demographic Scope: The paper focuses primarily on gender bias. While the authors suggest the method is principled, it remains to be seen how well it scales to more complex, intersecting biases like race, religion, or age.
  • Model Scale: The experiments were conducted on medium-sized models (BERT and RoBERTa). Testing this on massive Large Language Models (LLMs) with billions of parameters is a necessary next step.
  • Word List Dependency: The method currently relies on a predefined list of gender-sensitive words. Future work could involve automatically identifying which parts of the vocabulary need "collapsing" to ensure fairness.

Summary updated

Abstract

To mitigate societal biases implicitly encoded in recent successful pretrained language models, a diverse array of approaches have been proposed to encourage model fairness, focusing on prompting, data augmentation, regularized fine-tuning, and more. Despite the development, it is nontrivial to reach a principled understanding of fairness and an effective algorithm that can consistently debias language models. In this work, by rigorous evaluations of Neural Collapse -- a learning phenomenon happen in last-layer representations and classifiers in deep networks -- on fairness-related words, we find that debiased language models exhibit collapsed alignment between token representations and word embeddings. More importantly, this observation inspires us to design a principled fine-tuning method that can effectively improve fairness in a wide range of debiasing methods, while still preserving the performance of language models on standard natural language understanding tasks. We attach our code at https://github.com/Xujxyang/Fairness-NC-main
BibTeX
@inproceedings{xu2025collapsed,
  author = {Jingxuan Xu and Wuyang Chen and Linyi Li and Yao Zhao and Yunchao Wei},
  title = {{Collapsed Language Models Promote Fairness}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=kynD1UUk6q},
}
Figure