Abstract
Selecting the optimal pre-trained model (PTM) for IoT deployment involves balancing predictive performance with strict hardware constraints. Current recommendation systems often ignore hardware specifications, leading to "silent failures" where models are too resource-intensive for edge devices. This paper proposes two hardware-aware extensions to the Model Spider framework—Fusion and Shadow—to integrate hardware metrics directly into the model selection process.
Introduction: The Hardware-Blind Spot in AI Deployment
In modern deep learning, the standard workflow involves selecting a Pre-Trained Model (PTM) and fine-tuning it for a target task. While effective for cloud-based applications, this approach presents a significant challenge for IoT developers. A model with high benchmark accuracy may be operationally unfeasible due to excessive inference time, memory usage, or power consumption on edge hardware.
Existing recommendation systems, such as LogME or the original Model Spider, focus primarily on task suitability—predicting if a model is "smart" enough for the data. However, they remain "hardware-blind," failing to account for the physical constraints of microcontrollers and edge processors. This oversight leads to "performance bugs," where models are mathematically sound but practically undeployable.
Proposed Methodology: Hardware-Aware Model Selection
To bridge this gap, the authors propose evolving the Model Spider framework into a hardware-conscious system. Model Spider typically uses multi-head attention to match model "tokens" to task "tokens." The proposed extensions introduce hardware specifications as a primary input.
1. Model Spider Fusion
The Fusion approach treats hardware as a first-class citizen. It utilizes a Hardware Extractor ($\Psi_h$) to encode specifications like CPU type, RAM capacity, and I/O speed into a hardware token. This token is fused with the task token, enabling the recommendation engine to learn direct correlations between specific hardware profiles and model performance.
2. Model Spider Shadow
The Shadow approach employs a dual-ranking strategy. One pipeline ranks models based on task suitability (e.g., accuracy), while a parallel "shadow" pipeline ranks them based on hardware compatibility (e.g., latency, energy). These conflicting rankings are then aggregated using Copeland’s method, a social choice theory algorithm, to produce a single, balanced recommendation.
Evaluation Framework and Metrics
The paper outlines a rigorous procedure for establishing "ground truth" performance data. The authors advocate for a holistic evaluation that spans two categories:
- Hardware Metrics: Execution time, memory utilization, power consumption, CPU temperature, and carbon footprint.
- Model Metrics: Accuracy, precision, recall, and F1 score.
A key technical insight is the necessity of "stabilizing" hardware—ensuring CPU and RAM utilization reach nominal levels before benchmarking—to avoid data skew from transient background processes.
Key Insights: Toward Sustainable Edge AI
The proposed framework introduces several novel concepts for the IoT community:
- Hardware as an Embedding: Rather than using hardware specs as a simple binary filter, they are treated as latent representations that can be learned alongside task features.
- Sustainability as a Metric: By incorporating carbon footprint and energy consumption, the system aligns model selection with "Green AI" initiatives.
- Tunable Objectives: Engineers can weight their priorities (e.g., prioritizing energy efficiency over accuracy) through a mathematical objective function that balances model performance against normalized hardware costs.
Conclusion and Future Directions
This research provides a roadmap for the next generation of PTM recommenders. While currently focused on classification and regression, future work aims to address the complexities of Generative AI and Large Language Models (LLMs) on the edge. Additionally, the authors plan to explore how deep learning compilers (like ONNX) impact these recommendations, as software-level optimizations can significantly alter a model's performance profile on specific silicon.
Summary updated