Abstract
Graft is a first-of-its-kind inference serving system designed to optimize hybrid deep learning, where model execution is split between mobile devices and edge servers. Traditional hybrid systems suffer from "misalignment"—where varying network conditions cause devices to partition models at different layers, preventing efficient GPU batching. Graft introduces DNN re-alignment, a technique that restructures mismatched model fragments to share common layers. This enables massive request batching and fine-grained GPU sharing via NVIDIA CUDA MPS. Experimental results show that Graft reduces server resource consumption by up to 70% while maintaining strict Service-Level Objective (SLO) guarantees.
The Challenge: Misalignment in Hybrid Deep Learning
Deep Learning (DL) applications on mobile devices, such as augmented reality or voice assistants, often struggle with the trade-off between local execution (limited by hardware) and full cloud offloading (limited by latency and battery). Hybrid Deep Learning addresses this by partitioning a Deep Neural Network (DNN): the mobile device executes the initial layers, and the edge server processes the remainder.
However, network conditions like 5G signal strength are highly dynamic. To maintain latency targets, different mobile devices must partition the same model at different layers based on their current bandwidth. This creates a "misalignment" of fragments on the server side.
Standard inference systems treat these misaligned fragments as independent tasks. This prevents request batching, the primary mechanism for maximizing GPU efficiency. Consequently, edge servers waste significant GPU resources running numerous small, fragmented tasks.
The Graft Solution: DNN Re-alignment
Graft overcomes misalignment through a three-stage scheduling and execution pipeline:
- DNN Fragment Merging: The system identifies structurally identical fragments with similar latency budgets. It merges them into single instances, leveraging the discrete nature of GPU resource consumption where small throughput increases often don't require additional hardware.
- DNN Fragment Grouping: For heterogeneous fragments that cannot be directly merged, Graft clusters them using a balanced graph partitioning approach. This grouping is based on Euclidean distance across partition points, time budgets, and required throughput.
- DNN Re-alignment & Resource Allocation: This core mechanism re-partitions server-side fragments within a group. By splitting them again, Graft ensures they eventually feed into a single, shared "tail" fragment.
This shared tail (C' in Figure 2) can be batched across many users, significantly improving throughput. To manage these resources, Graft employs NVIDIA CUDA MPS (Multi-Process Service), allowing multiple processes to share a single GPU spatially by allocating specific percentages of GPU capacity.
Experimental Evaluation
The authors evaluated Graft using five representative DNN models (Inception-v3, ResNet-101, VGG11, Vision Transformer, and DeepLabV3) against real-world 5G network traces. The testbed utilized NVIDIA Jetson Nano and TX2 boards as mobile clients and high-end GPU servers for the edge.
Key Results
- Resource Efficiency: Graft improves resource efficiency by up to 70% compared to state-of-the-art systems like GSLICE.
- Throughput: Under resource constraints, Graft achieves 1.18x to 2.38x higher throughput than baselines due to its superior packing of requests.
- Latency Guarantees: Graft consistently meets latency SLOs. In some scenarios, it even reduces latency because shared fragments are executed more efficiently at higher batch sizes.
- Scalability: In large-scale simulations with thousands of fragments, Graft reduced resource consumption by 1.4x to 4x compared to static allocation.
Discussion and Future Work
While Graft significantly advances hybrid DL serving, some challenges remain. Frequent network fluctuations can lead to "re-alignment disruption," potentially mitigated by maintaining "shadow instances" during transitions. Additionally, the current system is optimized for users running the same model architecture. Future research will explore re-alignment across diverse architectures and handle potential precision mismatches (e.g., mixed INT8/FP16 quantization). Finally, extending these concepts to "Split Training" presents an interesting avenue for collaborative learning between clients and servers.
Summary updated