TL;DR
Graft is a first-of-its-kind inference serving system that optimizes "hybrid" deep learning—where a model is split between a mobile device and an edge server. By introducing a concept called DNN re-alignment, Graft restructures mismatched model fragments to share layers, enabling massive request batching and fine-grained GPU sharing that reduces server resource consumption by up to 70% while maintaining strict latency guarantees.
What Problem Does This Paper Address?
Deep Learning (DL) applications on mobile devices (like AR or voice assistants) often face a dilemma: mobile hardware is too weak for massive models, but offloading everything to the cloud can be slow and battery-draining. Hybrid Deep Learning solves this by partitioning a Deep Neural Network (DNN) into two parts: the mobile device runs the first few layers, and the server handles the rest.
However, there is a catch. Because network conditions (like 5G signal strength) fluctuate, different mobile devices will partition the same model at different layers. This results in a "misalignment" of fragments on the server side. Traditional inference systems treat these fragments as separate tasks, which prevents request batching—the industry-standard way to make GPUs efficient. Without batching, edge servers waste massive amounts of GPU power running tiny, fragmented tasks.
Methods
The authors propose Graft, a system designed to "re-align" these fragments. The core technical contribution is a three-stage scheduling pipeline:
- DNN Fragment Merging: It identifies fragments that are structurally identical and have similar latency budgets. It uses a "merging threshold" to combine them into single instances, exploiting the fact that GPU resource consumption is "discrete" (small increases in throughput don't always require more hardware).
- DNN Fragment Grouping: Since not all fragments can be merged, Graft groups heterogeneous fragments together. It treats this as a balanced graph partitioning problem, using Euclidean distance (based on partition points, time budgets, and throughput) to cluster similar fragments.
- DNN Re-alignment & Resource Allocation: This is the "secret sauce." For each group, Graft finds an optimal "re-partition point." It splits the server-side fragments again so that they all eventually feed into a single, shared "tail" fragment. This shared tail can then be batched across many different users.
To make this work on actual hardware, Graft utilizes NVIDIA CUDA MPS (Multi-Process Service), which allows multiple processes to share a single GPU spatially (by percentage of GPU share) rather than just through time-slicing.
Datasets
The researchers evaluated Graft using five representative DNN models from TorchVision:
- Inception-v3 & ResNet-101: Standard image classification.
- VGG11: A model with a simpler, linear architecture.
- Vision Transformer (ViT-B16): A modern, attention-based architecture.
- DeepLabV3 (MobileNetv3-Large): Used for semantic image segmentation.
They tested these against real-world 5G network traces to simulate the dynamic partitioning that happens in the wild. The hardware setup included NVIDIA Jetson Nano and TX2 boards as mobile clients and high-end GPU servers as the edge.
Key Insights and Ideas
- The Re-alignment "Aha!": The authors realized that even if fragments start at different layers, they are still part of the same model architecture. By forcing a second split on the server, you can create a "common denominator" fragment that is large enough to justify high-batch-size GPU execution.
- SLO-Driven Merging: Graft doesn't just merge for the sake of it. It respects Service-Level Objectives (SLOs). If merging two fragments would force a tight deadline on a fragment that previously had a relaxed one, Graft calculates if the batching gains outweigh the increased resource cost of meeting that tighter deadline.
- Heuristic Efficiency: Finding the "optimal" grouping is computationally expensive (combinatorial explosion). Graft uses a greedy heuristic that achieves within 0.7% of the optimal resource efficiency while being orders of magnitude faster to compute.
Key Results
- Resource Efficiency: Graft improves resource efficiency by up to 70% compared to state-of-the-art systems like GSLICE.
- Throughput: Under restricted resource conditions, Graft achieves 1.18x to 2.38x higher throughput than baseline systems because it can pack more requests into the same GPU space.
- Latency Guarantees: Despite the complex re-alignment and batching, Graft consistently meets latency SLOs. In some cases, it even achieves lower latency than baselines because the shared fragments are executed more efficiently.
- Scalability: In massive-scale simulations with thousands of fragments, Graft maintained a significant lead, reducing resource consumption by roughly 1.4x to 4x compared to standard static allocation.
Limitations and Future Work
- Realignment Overhead: Frequent network fluctuations can cause "re-alignment disruption," where the system spends too much time rescheduling. The authors suggest "shadow instances" (keeping old configurations running briefly) as a potential fix.
- Model Diversity: Currently, Graft works best when many users are running the same model (or models with shared backbones). Future work is needed to handle re-alignment across entirely different model architectures.
- Precision Mismatch: The paper assumes intermediate data is converted to full precision on the server. If different clients use different quantization levels (e.g., INT8 vs FP16), re-alignment becomes much more difficult.
- Split Training: The current system is for inference. Adapting this to "Split Training" (where clients and servers collaborate to train a model) would require handling much larger data transfers and memory requirements.
Summary updated