FairSMOE: Mitigating Multi-Attribute Fairness Problem with Sparse Mixture-of-Experts

Changdi Yang, Zheng Zhan, Ci Zhang, Yifan Gong, Yize Li, Zichong Meng, Jun Liu, Xuan Shen, Hao Tang, Geng Yuan, Pu Zhao, Xue Lin, Yanzhi WangIJCAI 2025 · DOI

Abstract

Multi-attribute fairness remains a significant challenge in machine learning, as real-world data often contains overlapping sensitive attributes such as age, gender, and anatomical site. Existing fairness-aware algorithms typically focus on a single attribute, failing to address the complexities of intersectional bias. We propose FairSMoE, a framework leveraging Sparse Mixture-of-Experts (SMoE) to achieve multi-attribute fairness without compromising predictive accuracy. By introducing Fairness-Guided Routing (FGR) and Fairness-driven Expert Management (FEM), FairSMoE dynamically allocates specialized experts to different sensitive attributes. Our evaluation on dermatology and facial recognition datasets demonstrates that FairSMoE achieves state-of-the-art performance in both fairness and accuracy.

Introduction

Machine learning models frequently exhibit biases toward specific sensitive groups. While single-attribute fairness is well-studied, multi-attribute scenarios introduce three primary hurdles: varying difficulty across attributes, inefficient resource allocation in static models, and "routing laziness" in standard SMoE architectures where routers become deterministic and under-utilize expert diversity. FairSMoE addresses these by treating multi-attribute fairness as a multi-task learning (MTL) problem, ensuring that models are fair across all dimensions without sacrificing performance.

Methodology

FairSMoE utilizes a two-pronged strategy within an MTL framework to optimize for both accuracy and fairness across multiple sensitive attributes.

Fairness-Guided Routing (FGR)

FGR employs a dual-branch structure to replace simple linear routers. A standard gating mechanism is augmented with a Sensitive Attribute (SA) branch that explicitly predicts attribute classes. To ensure feature neutrality, a confusion loss is applied, encouraging representations that are indistinguishable regarding sensitive attributes while allowing the router to use this information for optimal expert selection.

Fairness-driven Expert Management (FEM)

This dynamic strategy monitors fairness scores, such as Predictive Quality Disparity (PQD), during training. Attributes with lower fairness scores are dynamically assigned additional experts, increasing the model's learning capacity for more challenging tasks. This prevents the "one-size-fits-all" limitation of standard architectures.

Expert utilization heatmaps across different layers and expert counts, illustrating the specialization achieved by FairSMoE's routing policy.

Multi-Task Objective

The total loss function balances predictive accuracy, fairness losses for each attribute, confusion loss for disentanglement, and expert specialization loss. This approach reduces gradient conflict between tasks, minimizing the "tug-of-war" often seen in multi-objective optimization.

Experiments and Results

We evaluated FairSMoE on two primary benchmarks: the ISIC2019 dermatology dataset (attributes: age, sex, site) and the CelebA facial dataset (attributes: chubby, goatee, gender). We utilized DeiT and Swin Transformers as backbone architectures.

Performance Comparison

FairSMoE consistently outperformed state-of-the-art methods like Muffin, InfoFair, and MultiFair. On the ISIC2019 dataset with a DeiT-Small backbone, FairSMoE achieved 77.25% accuracy and a PQD score of 0.801, significantly higher than the vanilla SMoE baseline (74.49% accuracy, 0.773 PQD).

Fairness scores (PQD) across different attributes (sex, age, site) and overall accuracy for various DeiT model scales.

Efficiency and Insights

Despite its performance gains, FairSMoE maintains high efficiency. The DeiT-Small variant uses only 23M activated parameters, outperforming much larger models (up to 173M parameters). Key insights from our analysis include:

  • Mitigating Routing Laziness: FGR forces meaningful expert specialization, whereas vanilla routers often ignore attribute diversity.
  • Addressing Attribute Inequality: Our results confirm that fairness is naturally harder to achieve for certain attributes (e.g., age vs. gender); FEM's dynamic allocation effectively compensates for this disparity.

Conclusion

FairSMoE provides a robust and efficient solution for the multi-attribute fairness problem by leveraging the dynamic capacity of Sparse Mixture-of-Experts. While the method currently requires explicit attribute labels during training, it sets a new standard for balancing accuracy and intersectional fairness in computer vision tasks. Future work will explore extending these principles to large language models and tabular data.

Summary updated

BibTeX
@inproceedings{yang2025fairsmoe,
  author = {Changdi Yang and Zheng Zhan and Ci Zhang and Yifan Gong and Yize Li and Zichong Meng and Jun Liu and Xuan Shen and Hao Tang and Geng Yuan and Pu Zhao and Xue Lin and Yanzhi Wang},
  title = {{FairSMOE: Mitigating Multi-Attribute Fairness Problem with Sparse Mixture-of-Experts}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/69},
}
Figure