AMP4EC: Adaptive Model Partitioning Framework for Efficient Deep Learning Inference in Edge Computing Environments

Guilin Zhang, Wulan Guo, Ziqi Tan, Hailong Jiang

Abstract

Edge computing facilitates deep learning in resource-constrained environments, but challenges such as resource heterogeneity and dynamic constraints persist. This paper introduces AMP4EC, an Adaptive Model Partitioning framework designed to optimize deep learning inference in edge environments through real-time resource monitoring, dynamic model partitioning, and adaptive task scheduling. AMP4EC features a resource-aware model partitioner that splits deep learning models based on device capabilities, a task scheduler that ensures efficient load balancing using a weighted scoring mechanism, and a Docker-based deployment environment for validation. Experimental results show up to a 78% reduction in latency and a 414% improvement in throughput compared to baseline methods. The framework achieves consistent performance with low scheduling overhead across varying resource profiles, demonstrating adaptability in high-resource (1 CPU, 1GB RAM) and low-resource (0.4 CPU, 512MB RAM) scenarios. These results highlight AMP4EC's scalability, efficiency, and robustness for real-world edge deployments, addressing the critical need for efficient distributed inference in dynamic, resource-constrained environments.

Introduction

Edge computing brings AI processing closer to data sources, such as IoT sensors and mobile devices, to reduce latency and save network bandwidth. However, edge devices are typically resource-constrained, making it difficult to run computationally heavy Deep Neural Networks (DNNs). While Distributed Deep Learning (DDL) can partition a large model across multiple smaller devices, existing partitioning methods are mostly static. They assume a stable environment and fail to adapt when new devices join the network or available memory fluctuates. AMP4EC addresses this gap by introducing a real-time, adaptive approach to model partitioning and task scheduling.

Methodology

AMP4EC (Adaptive Model Partitioning for Edge Computing) relies on four interacting components to dynamically manage workloads:

  1. Resource Monitor: Continuously tracks CPU utilization, memory usage, and network I/O across all edge nodes in real-time using Docker's stats API.
  2. Model Partitioner: Utilizes a Resource-Aware Layerwise Optimization Strategy (RALOS). It analyzes the model layer-by-layer, estimates computational costs, and calculates partition boundaries that match the real-time hardware capabilities of available nodes.
  3. Task Scheduler: Uses a custom Node Selection Algorithm (NSA) to assign tasks based on a weighted scoring mechanism: 50% load balance, 20% resource availability, 20% current load, and 10% historical performance.
  4. Model Deployer: Manages the distribution, deployment, and teardown of partitioned sub-models inside lightweight Docker containers.
The AMP4EC framework architecture, illustrating the interaction between the Resource Monitor, Model Partitioner, Task Scheduler, and Model Deployer.

Experimental Evaluation

The framework was evaluated using a pre-trained MobileNetV2 model. The system's performance was benchmarked by sending batches of inference requests to a cluster of Docker containers configured to mimic edge devices with varying resource profiles (High, Medium, and Low).

Key insights from the evaluation revealed that memory constraints impact edge inference performance much more severely than CPU limitations. Additionally, by adding a caching layer to the task scheduler, AMP4EC significantly reduced network bandwidth requirements for repeated inference patterns. The heavy weighting on load balancing (50% of the scheduling score) ensured that work was distributed fairly, preventing bottlenecks and maintaining system speed.

Results

AMP4EC was compared against a monolithic baseline running the entire model on a single 2-core, 2GB RAM container. The results demonstrate the framework's superior efficiency in resource-constrained environments.

Comparative performance metrics of AMP4EC (with and without caching) against a monolithic baseline, showing significant improvements in latency and throughput.

As shown in the data, AMP4EC with caching achieved a 78.35% reduction in inference latency (dropping from 1082.53ms to 234.56ms) and a 414.73% improvement in throughput. The scheduling algorithm is lightweight, introducing only ~10ms of overhead, which is less than 2% of end-to-end latency. Despite the complexity of distributed deployment, the system maintained a tiny memory footprint of just 1.625MB.

Conclusion and Future Work

AMP4EC demonstrates that adaptive partitioning is essential for efficient edge AI in dynamic environments. While the current framework supports PyTorch and pre-deployment adaptation, future work will focus on integrating Kubernetes for cloud-to-edge orchestration, supporting multi-model deployments, and exploring privacy-preserving partitioning techniques. The team also aims to implement predictive resource allocation to stay ahead of network fluctuations.

Summary updated

BibTeX
@inproceedings{zhang2025amp,
  author = {Guilin Zhang and Wulan Guo and Ziqi Tan and Hailong Jiang},
  title = {{AMP4EC: Adaptive Model Partitioning Framework for Efficient Deep Learning Inference in Edge Computing Environments}},
  booktitle = {Fmec},
  year = {2025},
  doi = {10.1109/fmec65595.2025.11119384},
}
Figure