Recommending Pre-Trained Models for IoT Devices

Parth V. Patil, Wenxin Jiang, Huiyun Peng, Daniel Lugo, Kelechi G. Kalu, David J. LeBlanc, Lawrence Smith, Hyeonwoo Heo, Nathanael Aou, James C. Davis

Abstract

Selecting the optimal pre-trained model (PTM) for IoT deployment involves balancing predictive performance with strict hardware constraints. Current recommendation systems often ignore hardware specifications, leading to "silent failures" where models are too resource-intensive for edge devices. This paper proposes two hardware-aware extensions to the Model Spider framework—Fusion and Shadow—to integrate hardware metrics directly into the model selection process.

Introduction: The Hardware-Blind Spot in AI Deployment

In modern deep learning, the standard workflow involves selecting a Pre-Trained Model (PTM) and fine-tuning it for a target task. While effective for cloud-based applications, this approach presents a significant challenge for IoT developers. A model with high benchmark accuracy may be operationally unfeasible due to excessive inference time, memory usage, or power consumption on edge hardware.

Existing recommendation systems, such as LogME or the original Model Spider, focus primarily on task suitability—predicting if a model is "smart" enough for the data. However, they remain "hardware-blind," failing to account for the physical constraints of microcontrollers and edge processors. This oversight leads to "performance bugs," where models are mathematically sound but practically undeployable.

Current PTM recommendation pipelines often ignore hardware specifications, leading to deployment failures on resource-constrained devices.

Proposed Methodology: Hardware-Aware Model Selection

To bridge this gap, the authors propose evolving the Model Spider framework into a hardware-conscious system. Model Spider typically uses multi-head attention to match model "tokens" to task "tokens." The proposed extensions introduce hardware specifications as a primary input.

1. Model Spider Fusion

The Fusion approach treats hardware as a first-class citizen. It utilizes a Hardware Extractor ($\Psi_h$) to encode specifications like CPU type, RAM capacity, and I/O speed into a hardware token. This token is fused with the task token, enabling the recommendation engine to learn direct correlations between specific hardware profiles and model performance.

2. Model Spider Shadow

The Shadow approach employs a dual-ranking strategy. One pipeline ranks models based on task suitability (e.g., accuracy), while a parallel "shadow" pipeline ranks them based on hardware compatibility (e.g., latency, energy). These conflicting rankings are then aggregated using Copeland’s method, a social choice theory algorithm, to produce a single, balanced recommendation.

Proposed hardware-aware architectures: Model Spider Fusion (left) and Model Spider Shadow (right).

Evaluation Framework and Metrics

The paper outlines a rigorous procedure for establishing "ground truth" performance data. The authors advocate for a holistic evaluation that spans two categories:

  • Hardware Metrics: Execution time, memory utilization, power consumption, CPU temperature, and carbon footprint.
  • Model Metrics: Accuracy, precision, recall, and F1 score.

A key technical insight is the necessity of "stabilizing" hardware—ensuring CPU and RAM utilization reach nominal levels before benchmarking—to avoid data skew from transient background processes.

Key Insights: Toward Sustainable Edge AI

The proposed framework introduces several novel concepts for the IoT community:

  • Hardware as an Embedding: Rather than using hardware specs as a simple binary filter, they are treated as latent representations that can be learned alongside task features.
  • Sustainability as a Metric: By incorporating carbon footprint and energy consumption, the system aligns model selection with "Green AI" initiatives.
  • Tunable Objectives: Engineers can weight their priorities (e.g., prioritizing energy efficiency over accuracy) through a mathematical objective function that balances model performance against normalized hardware costs.

Conclusion and Future Directions

This research provides a roadmap for the next generation of PTM recommenders. While currently focused on classification and regression, future work aims to address the complexities of Generative AI and Large Language Models (LLMs) on the edge. Additionally, the authors plan to explore how deep learning compilers (like ONNX) impact these recommendations, as software-level optimizations can significantly alter a model's performance profile on specific silicon.

Summary updated

BibTeX
@inproceedings{patil2025recommending,
  author = {Parth V. Patil and Wenxin Jiang and Huiyun Peng and Daniel Lugo and Kelechi G. Kalu and David J. LeBlanc and Lawrence Smith and Hyeonwoo Heo and Nathanael Aou and James C. Davis},
  title = {{Recommending Pre-Trained Models for IoT Devices}},
  booktitle = {Nier@icse},
  year = {2025},
  doi = {10.1109/icse-nier66352.2025.00031},
}
Figure