Beyond Attribute-Centric Fairness: Modelling and Mitigating Unified Bias Representation in Omni-modal Large Language Models

Runqi Lin, Suqin Yuan, Zihao Fu, Chris RussellarXiv 2026 · arXiv

TL;DR

OmniDebias shifts the fairness paradigm for omni-modal large language models (OLLMs) from "attribute-centric" to "bias-centric." By disentangling social stereotypes from legitimate knowledge and consolidating them into a single, shared bias representation, it enables unified debiasing across multiple modalities and generalizes to unseen sensitive attributes without requiring separate, conflicting objectives.

What Problem Does This Paper Address?

As AI moves toward Omni-modal Large Language Models (OLLMs)—models that handle text, images, and more in a single joint embedding space—fairness becomes a scaling nightmare. Traditional "attribute-centric" debiasing requires a separate fairness objective for every combination of attribute (gender, race, age) and modality (text-to-image, image-to-text).

This approach has three major flaws:

  1. Scalability: It’s impossible to enumerate every possible demographic concept and interaction.
  2. Mutual Interference: Optimizing for one attribute (e.g., gender) can inadvertently degrade fairness for another (e.g., race) because their representations are entangled.
  3. Generalization: These methods fail on "out-of-distribution" (OOD) attributes that weren't explicitly targeted during training.
Figure 1. Schematic comparison of attribute-centric (left panel) and bias-centric (right panel) debiasing paradigms, showing their workflows (top row) and feature-space geometry (bottom row). Attribute-centric methods replace bias-contaminated directions with predefined unbiased ones, requiring target-specific priors that limit generalisation to unseen attributes. Bias-centric methods instead disentangle and mitigate shared bias while preserving legitimate attribute-related knowledge, enabling unified debiasing across attributes and modalities and facilitating generalisation to unseen attributes.

Methods

The authors introduce OmniDebias, which treats bias as a structured, shared phenomenon rather than a collection of independent problems.

1. Bias Disentanglement

The core idea is to exploit activation inconsistencies. Ideally, switching an attribute (e.g., male $\leftrightarrow$ female) should produce the same "shift" in the model's internal activations regardless of the context. However, in biased models, the shift for "CEO" looks different than the shift for "nurse" because stereotypes are entangled with the occupation's representation. By measuring these deviations across many contexts, OmniDebias isolates the "bias" component from "legitimate knowledge."

2. Bias Unification

Instead of treating gender bias and racial bias as separate vectors, the method hierarchically aggregates these extracted biases into a unified bias representation ($\Phi^{(l)}$). This captures the underlying "cross-attribute structure" of how the model encodes stereotypes.

3. Mitigation

This unified representation can be used in two ways:

  • OmniDebias-PE (Post-hoc Editing): A training-free method that projects model weights onto the orthogonal complement of the bias subspace at inference time.
  • OmniDebias-FT (Fine-tuning): An adaptive approach that dynamically extracts and penalizes bias-aligned components during training without needing ground-truth labels.

Datasets

The researchers evaluated their method on two representative OLLMs (Janus-Pro-7B and BAGEL-7B-MoT) using:

  • Extended WinoBias: 114 professions paired with gender, race, and age attributes for Text-to-Text (T2T) and Text-to-Image (T2I) tasks.
  • CMSC Dataset: 18 social concepts (personality, education, etc.) for Image-to-Text (I2T) description.
  • Socioeconomic Status (SES): Used as an unseen/OOD attribute to test if the model could generalize fairness to a concept it wasn't specifically "debiased" for.

Key Insights and Ideas

  • Bias has a "Shared Signature": The most striking insight is that different social biases (gender, race, age) share a common underlying structure in the model's latent space. Capturing this "unified bias" is more effective than chasing individual attributes.
  • Knowledge Preservation: By focusing on the inconsistency of shifts across contexts, the method avoids "throwing the baby out with the bathwater." It removes the stereotypical associations while keeping the actual meaning of the attributes (e.g., the model still knows what "female" means, just not that "female" implies "homemaker").
  • Zero-Shot Fairness: Because the bias representation is unified, the model shows improved fairness on attributes it never saw during the debiasing process (like Socioeconomic Status).

Key Results

OmniDebias consistently outperformed attribute-centric baselines across 11 different attribute-modality settings.

  • Quantitative: On Janus-Pro-7B, OmniDebias-FT improved fairness metrics across the board while maintaining or even improving generation quality (measured by Perplexity and MUSIQ scores). For example, in T2I tasks, it significantly reduced demographic skew compared to the vanilla model.
  • Qualitative: As seen in Figure 4, while the vanilla model predominantly generates white, middle-aged men when prompted for a "CEO," OmniDebias produces a much more balanced and diverse set of images.
Qualitative T2I results for the prompt "Generate a photo of a face of a CEO".

Limitations and Future Work

  • Coarse Categories: The current study uses relatively coarse demographic categories (e.g., binary gender). Real-world identities are much more granular (non-binary, gender-fluid), which may present more complex bias structures.
  • Modality Expansion: While it covers text and images, future OLLMs will include audio and video. Extending OmniDebias to these high-dimensional temporal modalities is a key next step.
  • Architectural Heterogeneity: The method works best on models with shared autoregressive representations. Models with highly specialized, modality-specific decoders (like certain flow-based image generators) might require further adaptation.

Summary updated

Abstract

Omni-modal large language models (OLLMs) provide joint multimodal understanding and generation capabilities, making them well-suited to complex environments and promising backbones for next-generation AI. Despite these advances, the harmful stereotypes encoded in such models manifest in increasingly diverse forms, leading to broader and more severe societal consequences. Existing debiasing methods typically impose separate fairness objectives for each sensitive attribute and modality to minimise demographic-dependent variations. However, in omni-modal settings, this attribute-centric paradigm struggles to jointly address diverse social biases with distinct fairness objectives and to generalise beyond predefined attributes. To this end, we propose OmniDebias, one of the first methods for promoting fairness in OLLMs, which employs a bias-centric formulation to extract and mitigate a shared bias representation across attributes and modalities while preserving attribute-related knowledge. Specifically, we exploit activation inconsistencies of a given attribute (e.g., gender) across semantic contexts (e.g., occupations) to disentangle bias from legitimate knowledge. We further consolidate multiple extracted biases into a shared representation, formulating a unified objective that captures the underlying structure across social biases and generalises to unseen attributes. Extensive experiments demonstrate that eliminating unified bias with both post-hoc editing and fine-tuning can consistently improve fairness across diverse attributes and modalities in OLLMs while maintaining favourable utility.
BibTeX
@misc{lin2026beyond,
  author = {Runqi Lin and Suqin Yuan and Zihao Fu and Chris Russell},
  title = {{Beyond Attribute-Centric Fairness: Modelling and Mitigating Unified Bias Representation in Omni-modal Large Language Models}},
  year = {2026},
  eprint = {2611.22889},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22889},
}
Figure