MoENAS: Mixture-of-Expert based Neural Architecture Search for jointly Accurate, Fair, and Robust Edge Deep Neural Networks

Lotfi Abdelkrim Mecharbat, Alberto Marchisio, Muhammad Shafique, Mohammad M. Ghassemi, Tuka AlhanaiarXiv 2025 · DOI · arXiv

Introduction

The proliferation of Deep Neural Networks (DNNs) on edge devices, such as smartphones and IoT sensors, has traditionally been driven by a singular focus: the trade-off between accuracy and computational efficiency. While this has led to highly optimized models, it often neglects critical dimensions of model performance, specifically fairness and robustness. This paper introduces MoENAS, a framework that integrates these ethical and reliability metrics directly into the Neural Architecture Search (NAS) process.

Preliminary analysis of state-of-the-art (SOTA) edge models reveals a significant "fairness gap." For instance, there is an average accuracy disparity of over 14% between the lightest and darkest skin tones in person classification tasks. Furthermore, these models frequently exhibit sensitivity to environmental factors like lighting and tend to overfit to training data. MoENAS addresses these issues by treating fairness, robustness, and generalization as primary objectives alongside accuracy.

Methodology

The MoENAS framework is built upon three core pillars designed to enhance model adaptability and efficiency.

1. Switch FFN (S-FFN) Layers

The central architectural innovation is the replacement of standard Feed-Forward Network (FFN) layers with a Mixture-of-Experts (MoE) structure, termed "Switch FFN." In this setup, a "router" module learns to dynamically dispatch input features to a specific subset of specialized experts. This mechanism allows the model to adapt its internal representations based on the input characteristics, such as varying skin tones or lighting conditions.

Comparison of a typical attention block and the proposed Switch FFN architecture.

2. Multi-Objective Hardware-Aware NAS

MoENAS utilizes a Hardware-aware Neural Architecture Search (HW-NAS) to explore a search space based on the MobileNetV2 macro-architecture. The search process employs a modified Bayesian Optimization (BO) strategy with an XGBoost surrogate model. Unlike traditional NAS, which might only optimize for accuracy and FLOPs, MoENAS concurrently optimizes four distinct objectives: Test Accuracy, Skin Fairness (measured on the Monk Skin Tone scale), Robustness to Light, and Generalization (minimizing the validation-test accuracy gap).

3. Expert Pruning for Edge Efficiency

To ensure that the resulting MoE models remain suitable for edge deployment, the authors implement a post-training expert pruning step. By analyzing expert usage across a validation set, the framework identifies and removes underutilized experts. This iterative process reduces the total parameter count while preserving the gains in fairness and accuracy, ensuring the final architecture is both compact and high-performing.

Key Insights

The research highlights that fairness is not merely a data-centric issue but is deeply tied to model architecture. Traditional scaling of model width or depth can inadvertently amplify biases present in the training data. By contrast, the MoE approach allows the network to allocate dedicated parameters to "minority" cases without them being overshadowed by the majority data. Interestingly, the router naturally learns to cluster images by demographic features, suggesting that experts specialize in specific environmental or physical attributes.

Experimental Results

The MoENAS models (S, XS, and XXS) were evaluated against SOTA edge models using the COCO and FACET datasets. The results demonstrate a significant shift in the Pareto-optimal frontier for edge-ready models.

Pareto front analysis of Test Accuracy versus Skin Fairness for MoENAS and SOTA edge models.

Key performance improvements include:

  • Accuracy: A 4.02% improvement over existing SOTA edge DNNs.
  • Fairness: A reduction in skin tone accuracy disparity from 14.09% to just 5.60%.
  • Robustness: A 3.80% increase in reliability across varying lighting conditions.
  • Generalization: An extremely low overfitting gap of 0.21%.

The MoENAS-S model, for example, maintains a manageable size of approximately 2.7M parameters, proving that ethical and robust AI does not require prohibitive computational overhead.

Conclusion

MoENAS demonstrates that fairness and robustness can be effectively integrated into the design of edge-native DNNs through Mixture-of-Experts and multi-objective NAS. While the increased parameter count of MoE models presents a challenge for devices with extreme memory constraints, the framework provides a robust path forward for developing more equitable and reliable AI systems. Future work will explore the application of MoENAS to other sensitive domains, such as medical imaging and autonomous driving.

Summary updated

Abstract

There has been a surge in optimizing edge Deep Neural Networks (DNNs) for accuracy and efficiency using traditional optimization techniques such as pruning, and more recently, employing automatic design methodologies. However, the focus of these design techniques has often overlooked critical metrics such as fairness, robustness, and generalization. As a result, when evaluating SOTA edge DNNs' performance in image classification using the FACET dataset, we found that they exhibit significant accuracy disparities (14.09%) across 10 different skin tones, alongside issues of non-robustness and poor generalizability. In response to these observations, we introduce Mixture-of-Experts-based Neural Architecture Search (MoENAS), an automatic design technique that navigates through a space of mixture of experts to discover accurate, fair, robust, and general edge DNNs. MoENAS improves the accuracy by 4.02% compared to SOTA edge DNNs and reduces the skin tone accuracy disparities from 14.09% to 5.60%, while enhancing robustness by 3.80% and minimizing overfitting to 0.21%, all while keeping model size close to state-of-the-art models average size (+0.4M). With these improvements, MoENAS establishes a new benchmark for edge DNN design, paving the way for the development of more inclusive and robust edge DNNs.
BibTeX
@misc{mecharbat2025moenas,
  author = {Lotfi Abdelkrim Mecharbat and Alberto Marchisio and Muhammad Shafique and Mohammad M. Ghassemi and Tuka Alhanai},
  title = {{MoENAS: Mixture-of-Expert based Neural Architecture Search for jointly Accurate, Fair, and Robust Edge Deep Neural Networks}},
  year = {2025},
  eprint = {2502.07422},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.07422},
}
Figure