Abstract
Edge computing facilitates deep learning in resource-constrained environments, but challenges such as resource heterogeneity and dynamic constraints persist. This paper introduces AMP4EC, an Adaptive Model Partitioning framework designed to optimize deep learning inference in edge environments through real-time resource monitoring, dynamic model partitioning, and adaptive task scheduling. AMP4EC features a resource-aware model partitioner that splits deep learning models based on device capabilities, a task scheduler that ensures efficient load balancing using a weighted scoring mechanism, and a Docker-based deployment environment for validation. Experimental results show up to a 78% reduction in latency and a 414% improvement in throughput compared to baseline methods. The framework achieves consistent performance with low scheduling overhead across varying resource profiles, demonstrating adaptability in high-resource (1 CPU, 1GB RAM) and low-resource (0.4 CPU, 512MB RAM) scenarios. These results highlight AMP4EC's scalability, efficiency, and robustness for real-world edge deployments, addressing the critical need for efficient distributed inference in dynamic, resource-constrained environments.
Introduction
Edge computing brings AI processing closer to data sources, such as IoT sensors and mobile devices, to reduce latency and save network bandwidth. However, edge devices are typically resource-constrained, making it difficult to run computationally heavy Deep Neural Networks (DNNs). While Distributed Deep Learning (DDL) can partition a large model across multiple smaller devices, existing partitioning methods are mostly static. They assume a stable environment and fail to adapt when new devices join the network or available memory fluctuates. AMP4EC addresses this gap by introducing a real-time, adaptive approach to model partitioning and task scheduling.
Methodology
AMP4EC (Adaptive Model Partitioning for Edge Computing) relies on four interacting components to dynamically manage workloads:
- Resource Monitor: Continuously tracks CPU utilization, memory usage, and network I/O across all edge nodes in real-time using Docker's stats API.
- Model Partitioner: Utilizes a Resource-Aware Layerwise Optimization Strategy (RALOS). It analyzes the model layer-by-layer, estimates computational costs, and calculates partition boundaries that match the real-time hardware capabilities of available nodes.
- Task Scheduler: Uses a custom Node Selection Algorithm (NSA) to assign tasks based on a weighted scoring mechanism: 50% load balance, 20% resource availability, 20% current load, and 10% historical performance.
- Model Deployer: Manages the distribution, deployment, and teardown of partitioned sub-models inside lightweight Docker containers.
Experimental Evaluation
The framework was evaluated using a pre-trained MobileNetV2 model. The system's performance was benchmarked by sending batches of inference requests to a cluster of Docker containers configured to mimic edge devices with varying resource profiles (High, Medium, and Low).
Key insights from the evaluation revealed that memory constraints impact edge inference performance much more severely than CPU limitations. Additionally, by adding a caching layer to the task scheduler, AMP4EC significantly reduced network bandwidth requirements for repeated inference patterns. The heavy weighting on load balancing (50% of the scheduling score) ensured that work was distributed fairly, preventing bottlenecks and maintaining system speed.
Results
AMP4EC was compared against a monolithic baseline running the entire model on a single 2-core, 2GB RAM container. The results demonstrate the framework's superior efficiency in resource-constrained environments.
As shown in the data, AMP4EC with caching achieved a 78.35% reduction in inference latency (dropping from 1082.53ms to 234.56ms) and a 414.73% improvement in throughput. The scheduling algorithm is lightweight, introducing only ~10ms of overhead, which is less than 2% of end-to-end latency. Despite the complexity of distributed deployment, the system maintained a tiny memory footprint of just 1.625MB.
Conclusion and Future Work
AMP4EC demonstrates that adaptive partitioning is essential for efficient edge AI in dynamic environments. While the current framework supports PyTorch and pre-deployment adaptation, future work will focus on integrating Kubernetes for cloud-to-edge orchestration, supporting multi-model deployments, and exploring privacy-preserving partitioning techniques. The team also aims to implement predictive resource allocation to stay ahead of network fluctuations.
Summary updated