Introduction
The proliferation of Deep Neural Networks (DNNs) on edge devices, such as smartphones and IoT sensors, has traditionally been driven by a singular focus: the trade-off between accuracy and computational efficiency. While this has led to highly optimized models, it often neglects critical dimensions of model performance, specifically fairness and robustness. This paper introduces MoENAS, a framework that integrates these ethical and reliability metrics directly into the Neural Architecture Search (NAS) process.
Preliminary analysis of state-of-the-art (SOTA) edge models reveals a significant "fairness gap." For instance, there is an average accuracy disparity of over 14% between the lightest and darkest skin tones in person classification tasks. Furthermore, these models frequently exhibit sensitivity to environmental factors like lighting and tend to overfit to training data. MoENAS addresses these issues by treating fairness, robustness, and generalization as primary objectives alongside accuracy.
Methodology
The MoENAS framework is built upon three core pillars designed to enhance model adaptability and efficiency.
1. Switch FFN (S-FFN) Layers
The central architectural innovation is the replacement of standard Feed-Forward Network (FFN) layers with a Mixture-of-Experts (MoE) structure, termed "Switch FFN." In this setup, a "router" module learns to dynamically dispatch input features to a specific subset of specialized experts. This mechanism allows the model to adapt its internal representations based on the input characteristics, such as varying skin tones or lighting conditions.
2. Multi-Objective Hardware-Aware NAS
MoENAS utilizes a Hardware-aware Neural Architecture Search (HW-NAS) to explore a search space based on the MobileNetV2 macro-architecture. The search process employs a modified Bayesian Optimization (BO) strategy with an XGBoost surrogate model. Unlike traditional NAS, which might only optimize for accuracy and FLOPs, MoENAS concurrently optimizes four distinct objectives: Test Accuracy, Skin Fairness (measured on the Monk Skin Tone scale), Robustness to Light, and Generalization (minimizing the validation-test accuracy gap).
3. Expert Pruning for Edge Efficiency
To ensure that the resulting MoE models remain suitable for edge deployment, the authors implement a post-training expert pruning step. By analyzing expert usage across a validation set, the framework identifies and removes underutilized experts. This iterative process reduces the total parameter count while preserving the gains in fairness and accuracy, ensuring the final architecture is both compact and high-performing.
Key Insights
The research highlights that fairness is not merely a data-centric issue but is deeply tied to model architecture. Traditional scaling of model width or depth can inadvertently amplify biases present in the training data. By contrast, the MoE approach allows the network to allocate dedicated parameters to "minority" cases without them being overshadowed by the majority data. Interestingly, the router naturally learns to cluster images by demographic features, suggesting that experts specialize in specific environmental or physical attributes.
Experimental Results
The MoENAS models (S, XS, and XXS) were evaluated against SOTA edge models using the COCO and FACET datasets. The results demonstrate a significant shift in the Pareto-optimal frontier for edge-ready models.
Key performance improvements include:
- Accuracy: A 4.02% improvement over existing SOTA edge DNNs.
- Fairness: A reduction in skin tone accuracy disparity from 14.09% to just 5.60%.
- Robustness: A 3.80% increase in reliability across varying lighting conditions.
- Generalization: An extremely low overfitting gap of 0.21%.
The MoENAS-S model, for example, maintains a manageable size of approximately 2.7M parameters, proving that ethical and robust AI does not require prohibitive computational overhead.
Conclusion
MoENAS demonstrates that fairness and robustness can be effectively integrated into the design of edge-native DNNs through Mixture-of-Experts and multi-objective NAS. While the increased parameter count of MoE models presents a challenge for devices with extreme memory constraints, the framework provides a robust path forward for developing more equitable and reliable AI systems. Future work will explore the application of MoENAS to other sensitive domains, such as medical imaging and autonomous driving.
Summary updated