Recommending Pre-Trained Models for IoT Devices

Parth V. Patil, Wenxin Jiang, Huiyun Peng, Daniel Lugo, Kelechi G. Kalu, Josh LeBlanc, Lawrence Smith, Hyeonwoo Heo, Nathanael Aou, James C. Davis

TL;DR

This paper addresses a critical blind spot in AI deployment: selecting pre-trained models (PTMs) for resource-constrained IoT devices. By proposing "hardware-aware" modifications to the Model Spider framework, the authors move beyond simple accuracy metrics to recommend models that actually fit within the CPU, memory, and energy budgets of edge hardware.

What Problem Does This Paper Address?

In the world of Deep Learning, we rarely start from scratch. Engineers typically grab a Pre-Trained Model (PTM) and fine-tune it for their specific needs. However, for IoT developers, this is a minefield. A model might boast 99% accuracy on a benchmark but take ten minutes to run a single inference on a microcontroller or drain a battery in an hour.

Existing recommendation systems like LogME or ModelSpider are great at predicting if a model is "smart" enough for a task, but they are "hardware-blind." They don't account for the "silent bugs" of performance—where a model is mathematically perfect but operationally impossible. This paper identifies two major gaps: the lack of IoT-specific inputs in recommendation pipelines and the absence of a "ground truth" ranking that compares how different models perform across diverse IoT hardware.

Methods

The authors propose evolving the existing Model Spider framework—which uses a multi-head attention mechanism to match model "tokens" to task "tokens"—into a hardware-conscious system. They introduce two primary architectural approaches:

  1. Model Spider Fusion: This approach treats hardware as a first-class citizen. It introduces a Hardware Extractor ($\Psi_h$) that encodes specifications (CPU type, RAM size, I/O speed) into a hardware token. This token is then fused with the task token, allowing the recommendation engine to learn correlations between specific hardware and model performance directly.
  2. Model Spider Shadow: This is a dual-ranking system. One pipeline ranks models based on task suitability (accuracy), while a parallel "shadow" pipeline ranks them based on hardware compatibility (speed, energy). The system then uses Copeland’s method—a social choice theory algorithm—to aggregate these conflicting rankings into a single, balanced recommendation.

To support these methods, the authors provide a tunable objective function that allows engineers to weight their priorities (e.g., "I care 70% about energy efficiency and 30% about accuracy").

Datasets

The paper functions as a foundational framework and research agenda rather than a report on a single static dataset. However, it outlines a rigorous Performance Evaluation Procedure (Algorithm 1) for creating the necessary ground-truth data.

The authors advocate for collecting metrics across two categories:

  • Hardware Metrics: Execution time, memory utilization, power consumption, CPU temperature, and carbon footprint.
  • Model Metrics: Accuracy, precision, recall, and F1 score.

They emphasize the need for "stabilizing" hardware (waiting for CPU/RAM utilization to hit nominal levels) before benchmarking to ensure the data isn't skewed by transient background processes.

Key Insights and Ideas

  • "Performance bugs are silent bugs": The authors highlight a poignant quote from practitioners: you often don't know a model is a failure until it's deployed and the hardware chokes. Recommendation systems must move "upstream" to prevent this.
  • Hardware as an Embedding: One of the most clever ideas is treating hardware specs not just as a filter (e.g., "must have < 1GB RAM") but as a latent representation that can be learned alongside task features.
  • Sustainability as a Metric: By including "Carbon Footprint" and "Energy Consumption" in their ranking system, the authors align model selection with the growing need for "Green AI" in the IoT space.

Key Results

As a "Vision and Research Agenda" paper, the primary results are the conceptual frameworks:

  • The Fusion and Shadow Architectures: These provide a roadmap for building the next generation of PTM recommenders.
  • The Weighted Ranking Formula: The authors provide a concrete mathematical objective function (Equation 1) that balances model performance $f(\alpha)$ against normalized hardware metrics $HW_i(\alpha)$.
  • Identification of Research Gaps: The paper formalizes four Research Questions (RQs) regarding parameter weighting, the impact of optimization (quantization/distillation), and correlations between hardware resources and model architecture.

Limitations and Future Work

The authors acknowledge that their current focus is on relatively "basic" tasks like image classification and regression. The "frontier" of AI—Generative AI and Large Language Models (LLMs)—presents a much steeper challenge for IoT hardware that hasn't been fully addressed yet.

Future work involves expanding the framework to handle complex model reuse scenarios, such as using PTMs as "backbones" for object detection or segmentation, where the hardware overhead of the "heads" and "decoders" added during fine-tuning must also be predicted. They also aim to bridge the gap between different deep learning compilers (like ONNX), which can significantly change a model's performance profile on specific silicon.

Summary updated

BibTeX
@inproceedings{patil2025recommending,
  author = {Parth V. Patil and Wenxin Jiang and Huiyun Peng and Daniel Lugo and Kelechi G. Kalu and Josh LeBlanc and Lawrence Smith and Hyeonwoo Heo and Nathanael Aou and James C. Davis},
  title = {{Recommending Pre-Trained Models for IoT Devices}},
  booktitle = {Nier@icse},
  year = {2025},
  doi = {10.1109/icse-nier66352.2025.00031},
}
Figure