Graft: Efficient Inference Serving for Hybrid Deep Learning With SLO Guarantees via DNN Re-Alignment

Jing Wu, Lin Wang, Qirui Jin, Fangming Liu

TL;DR

Graft is a first-of-its-kind inference serving system that optimizes "hybrid" deep learning—where a model is split between a mobile device and an edge server. By introducing a concept called DNN re-alignment, Graft restructures mismatched model fragments to share layers, enabling massive request batching and fine-grained GPU sharing that reduces server resource consumption by up to 70% while maintaining strict latency guarantees.

What Problem Does This Paper Address?

Deep Learning (DL) applications on mobile devices (like AR or voice assistants) often face a dilemma: mobile hardware is too weak for massive models, but offloading everything to the cloud can be slow and battery-draining. Hybrid Deep Learning solves this by partitioning a Deep Neural Network (DNN) into two parts: the mobile device runs the first few layers, and the server handles the rest.

However, there is a catch. Because network conditions (like 5G signal strength) fluctuate, different mobile devices will partition the same model at different layers. This results in a "misalignment" of fragments on the server side. Traditional inference systems treat these fragments as separate tasks, which prevents request batching—the industry-standard way to make GPUs efficient. Without batching, edge servers waste massive amounts of GPU power running tiny, fragmented tasks.

Methods

The authors propose Graft, a system designed to "re-align" these fragments. The core technical contribution is a three-stage scheduling pipeline:

  1. DNN Fragment Merging: It identifies fragments that are structurally identical and have similar latency budgets. It uses a "merging threshold" to combine them into single instances, exploiting the fact that GPU resource consumption is "discrete" (small increases in throughput don't always require more hardware).
  2. DNN Fragment Grouping: Since not all fragments can be merged, Graft groups heterogeneous fragments together. It treats this as a balanced graph partitioning problem, using Euclidean distance (based on partition points, time budgets, and throughput) to cluster similar fragments.
  3. DNN Re-alignment & Resource Allocation: This is the "secret sauce." For each group, Graft finds an optimal "re-partition point." It splits the server-side fragments again so that they all eventually feed into a single, shared "tail" fragment. This shared tail can then be batched across many different users.

To make this work on actual hardware, Graft utilizes NVIDIA CUDA MPS (Multi-Process Service), which allows multiple processes to share a single GPU spatially (by percentage of GPU share) rather than just through time-slicing.

Datasets

The researchers evaluated Graft using five representative DNN models from TorchVision:

  • Inception-v3 & ResNet-101: Standard image classification.
  • VGG11: A model with a simpler, linear architecture.
  • Vision Transformer (ViT-B16): A modern, attention-based architecture.
  • DeepLabV3 (MobileNetv3-Large): Used for semantic image segmentation.

They tested these against real-world 5G network traces to simulate the dynamic partitioning that happens in the wild. The hardware setup included NVIDIA Jetson Nano and TX2 boards as mobile clients and high-end GPU servers as the edge.

Key Insights and Ideas

  • The Re-alignment "Aha!": The authors realized that even if fragments start at different layers, they are still part of the same model architecture. By forcing a second split on the server, you can create a "common denominator" fragment that is large enough to justify high-batch-size GPU execution.
  • SLO-Driven Merging: Graft doesn't just merge for the sake of it. It respects Service-Level Objectives (SLOs). If merging two fragments would force a tight deadline on a fragment that previously had a relaxed one, Graft calculates if the batching gains outweigh the increased resource cost of meeting that tighter deadline.
  • Heuristic Efficiency: Finding the "optimal" grouping is computationally expensive (combinatorial explosion). Graft uses a greedy heuristic that achieves within 0.7% of the optimal resource efficiency while being orders of magnitude faster to compute.

Key Results

  • Resource Efficiency: Graft improves resource efficiency by up to 70% compared to state-of-the-art systems like GSLICE.
  • Throughput: Under restricted resource conditions, Graft achieves 1.18x to 2.38x higher throughput than baseline systems because it can pack more requests into the same GPU space.
  • Latency Guarantees: Despite the complex re-alignment and batching, Graft consistently meets latency SLOs. In some cases, it even achieves lower latency than baselines because the shared fragments are executed more efficiently.
  • Scalability: In massive-scale simulations with thousands of fragments, Graft maintained a significant lead, reducing resource consumption by roughly 1.4x to 4x compared to standard static allocation.

Limitations and Future Work

  • Realignment Overhead: Frequent network fluctuations can cause "re-alignment disruption," where the system spends too much time rescheduling. The authors suggest "shadow instances" (keeping old configurations running briefly) as a potential fix.
  • Model Diversity: Currently, Graft works best when many users are running the same model (or models with shared backbones). Future work is needed to handle re-alignment across entirely different model architectures.
  • Precision Mismatch: The paper assumes intermediate data is converted to full precision on the server. If different clients use different quantization levels (e.g., INT8 vs FP16), re-alignment becomes much more difficult.
  • Split Training: The current system is for inference. Adapting this to "Split Training" (where clients and servers collaborate to train a model) would require handling much larger data transfers and memory requirements.

Summary updated

BibTeX
@article{wu2024graft,
  author = {Jing Wu and Lin Wang and Qirui Jin and Fangming Liu},
  title = {{Graft: Efficient Inference Serving for Hybrid Deep Learning With SLO Guarantees via DNN Re-Alignment}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2024},
  doi = {10.1109/tpds.2023.3340518},
}
Figure