PockEngine: Sparse and Efficient Fine-tuning in a Pocket

Ligeng Zhu, Lanxiang Hu, Ji Lin, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, Song Han

TL;DR

PockEngine is a tiny, high-efficiency training engine designed to bring deep learning fine-tuning directly to edge devices like smartphones and microcontrollers. By utilizing "compilation-first" graph pruning and sparse backpropagation, it enables local adaptation of large models—including LlamaV2-7B—with up to 15× speedups and 21× memory savings compared to standard frameworks.

What Problem Does This Paper Address?

Deep learning is shifting toward personalization, but training models remains a "cloud-first" activity. Existing frameworks (PyTorch, TensorFlow) are optimized for massive GPU clusters and carry significant runtime overhead (often >300MB just for the host language).

Edge devices face two massive hurdles:

  1. Resource Limitations: Edge hardware has orders of magnitude less memory and compute power than cloud servers.
  2. Hardware Diversity: The "edge" ranges from ARM microcontrollers to Apple M1 chips and Qualcomm DSPs, each requiring different low-level optimizations.

Current edge libraries (like TF-Lite or TensorRT) are almost exclusively for inference. PockEngine fills this gap by providing a system-level solution for on-device training.

Methods

PockEngine departs from traditional "runtime-differentiation" (like PyTorch's Autograd) in favor of a compilation-based workflow.

  • Sparse Backpropagation (Sparse-BP): Instead of updating the entire model, PockEngine prunes the backward computation graph. It supports bias-only updates, layer-wise updates, and even sub-layer (channel-level) updates. This allows the system to release intermediate activations (the biggest memory hog in training) much earlier.
  • Compilation-First Autodiff: The entire training graph (forward, backward, and optimization) is derived at compile-time. This eliminates the Python runtime overhead and allows the compiler to perform aggressive dead-code elimination.
  • Training Graph Optimizations:
    • Operator Reordering: Gradients are applied to weights immediately during the backward pass rather than waiting for the end, significantly reducing peak memory.
  • Backend Switching: It can wrap "inference-only" vendor libraries (like Qualcomm’s SNPE or NVIDIA’s TensorRT) to perform training operations by treating backward passes as standard forward ops.
  • Winograd for Training: It applies Winograd convolution—usually an inference-only trick—to the frozen layers of a training graph to accelerate the overall process.

Datasets

The authors evaluated PockEngine across a wide variety of tasks:

  • Vision: ImageNet (pre-training), followed by fine-tuning on Cars, CIFAR-10, CUB, Flowers, Food-101, Pets, and Visual Wake Words (VWW).
  • NLP: BERT and DistilBERT evaluated on the GLUE benchmark (CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2).
  • Chatbots: LlamaV2-7B fine-tuned using the Stanford Alpaca dataset and evaluated via Alpaca-Eval and MT-Bench.

Key Insights and Ideas

  • The "Inference-Only" Myth: The paper proves that libraries designed only for inference can be used for training if you represent the backward pass as a static graph of primitive operations.
  • Strategic Freezing: For CNNs, updating the first convolution in each block is highly effective. For Transformers, the attention modules and the first linear layer in the Feed-Forward Network (FFN) are the most critical for adaptation.
  • System-Algorithm Co-design: Sparse-BP isn't just a mathematical trick to reduce FLOPs; it’s a system-level lever. By not needing to compute certain gradients, the engine can skip storing the corresponding "activations," which are the primary cause of Out-of-Memory (OOM) errors on edge devices.

Key Results

  • Speed: PockEngine achieved up to 15× speedup over off-the-shelf TensorFlow on a Raspberry Pi 4.
  • Memory: It demonstrated 5.6× to 21.3× memory savings during backpropagation. For example, MCUNet's memory footprint dropped from 3.6MB to just 173KB.
  • LLM Performance: On an NVIDIA Jetson AGX Orin, PockEngine fine-tuned LlamaV2-7B at 550 tokens/s, which is 7.9× faster than the PyTorch baseline.
  • Accuracy: Despite the heavy pruning, the sparse update schemes stayed within 1% accuracy of full fine-tuning across most vision and NLP tasks.

Limitations and Future Work

  • Offline Search: The "optimal" sparse update scheme (deciding which layers to freeze) currently requires an offline evolutionary search based on a cost model. Automating this search to be more dynamic or "on-the-fly" would be a significant next step.
  • Fine-tuning Focus: The engine is specifically optimized for fine-tuning/transfer learning. It is not intended for training large models from scratch on the edge, which remains computationally prohibitive.
  • Quantization: While the paper mentions DSP support (which often implies integer math), a deeper integration with quantization-aware training (QAT) on the edge could further reduce the resource floor.

Summary updated

Abstract

On-device learning and efficient fine-tuning enable continuous and privacy-preserving customization (e.g., locally fine-tuning large language models on personalized data). However, existing training frameworks are designed for cloud servers with powerful accelerators (e.g., GPUs, TPUs) and lack the optimizations for learning on the edge, which faces challenges of resource limitations and edge hardware diversity. We introduce PockEngine: a tiny, sparse and efficient engine to enable fine-tuning on various edge devices. PockEngine supports sparse backpropagation: it prunes the backward graph and sparsely updates the model with measured memory saving and latency reduction while maintaining the model quality. Secondly, PockEngine is compilation first: the entire training graph (including forward, backward and optimization steps) is derived at compile-time, which reduces the runtime overhead and brings opportunities for graph transformations. PockEngine also integrates a rich set of training graph optimizations, thus can further accelerate the training cost, including operator reordering and backend switching. PockEngine supports diverse applications, frontends and hardware backends: it flexibly compiles and tunes models defined in PyTorch/TensorFlow/Jax and deploys binaries to mobile CPU/GPU/DSPs. We evaluated PockEngine on both vision models and large language models. PockEngine achieves up to 15 $\times$ speedup over off-the-shelf TensorFlow (Raspberry Pi), 5.6 $\times$ memory saving back-propagation (Jetson AGX Orin). Remarkably, PockEngine enables fine-tuning LLaMav2-7B on NVIDIA Jetson AGX Orin at 550 tokens/s, 7.9$\times$ faster than the PyTorch.
BibTeX
@inproceedings{zhu2023pockengine,
  author = {Ligeng Zhu and Lanxiang Hu and Ji Lin and Wei-Ming Chen and Wei-Chen Wang and Chuang Gan and Song Han},
  title = {{PockEngine: Sparse and Efficient Fine-tuning in a Pocket}},
  booktitle = {MICRO},
  year = {2023},
  doi = {10.1145/3613424.3614307},
}

Loading related content...

Figure