PockEngine: Sparse and Efficient Fine-tuning in a Pocket

Ligeng Zhu, Lanxiang Hu, Ji Lin, Wei-Chen Wang, Wei-Ming Chen, Chuang Gan, Song Han

Abstract

On-device learning and efficient fine-tuning are essential for continuous, privacy-preserving customization of large models, such as Large Language Models (LLMs), on personalized data. However, traditional training frameworks are optimized for cloud-scale accelerators and lack the necessary adaptations for resource-constrained edge hardware. We present PockEngine, a sparse and efficient engine designed specifically for fine-tuning on diverse edge devices. PockEngine utilizes sparse backpropagation to prune the backward graph, significantly reducing memory overhead and latency while maintaining model quality. By adopting a "compilation-first" approach, PockEngine derives the entire training graph at compile-time, eliminating runtime overhead and enabling advanced graph transformations. Our evaluation demonstrates that PockEngine achieves up to 15× speedup over TensorFlow on Raspberry Pi and enables LlamaV2-7B fine-tuning at 550 tokens/s on NVIDIA Jetson AGX Orin, outperforming PyTorch by 7.9×.

Introduction

The paradigm of deep learning is shifting toward personalization, where models must adapt to individual user data locally to ensure privacy and low latency. Despite this shift, model training remains a "cloud-first" activity. Standard frameworks like PyTorch and TensorFlow are designed for massive GPU clusters; they carry significant runtime overhead (often exceeding 300MB for the host language alone) and lack the flexibility to handle the extreme resource limitations and hardware diversity of the edge.

Edge devices, ranging from ARM-based microcontrollers to mobile DSPs and Apple M-series chips, face two primary hurdles: limited memory/compute power and a fragmented hardware ecosystem. While libraries like TF-Lite and TensorRT have addressed edge inference, PockEngine fills the critical gap for on-device training.

System Design and Methodology

PockEngine departs from the traditional "runtime-differentiation" model in favor of a specialized compilation-based workflow.

Sparse Backpropagation (Sparse-BP)

Memory is the primary bottleneck for on-device training, largely due to the need to store intermediate activations for gradient calculation. PockEngine introduces Sparse-BP, which prunes the backward computation graph. By supporting bias-only, layer-wise, and sub-layer (channel-level) updates, the engine can release intermediate activations much earlier than standard frameworks. This system-algorithm co-design ensures that the reduction in FLOPs translates directly into significant memory savings.

Compilation-First Autodiff

Unlike frameworks that compute gradients dynamically at runtime, PockEngine derives the entire training graph—including forward, backward, and optimization steps—at compile-time. This approach eliminates Python runtime overhead and allows the compiler to perform aggressive dead-code elimination and graph-level optimizations.

Training Graph Optimizations

PockEngine integrates several novel optimizations to further accelerate the training process:

  • Operator Reordering: Gradients are applied to weights immediately during the backward pass. This prevents the system from holding multiple large gradient tensors in memory simultaneously, reducing peak memory usage.
  • Backend Switching: PockEngine can wrap "inference-only" vendor libraries (such as Qualcomm’s SNPE or NVIDIA’s TensorRT). It treats backward passes as standard forward operations, allowing it to leverage highly optimized proprietary kernels for training.
  • Winograd for Training: The engine applies Winograd convolution—typically an inference-only optimization—to the frozen layers of the training graph to accelerate the overall execution.

Experimental Evaluation

We evaluated PockEngine across a diverse set of tasks and hardware backends, including vision models, BERT-based NLP models, and Large Language Models.

Performance and Speed

PockEngine achieved a 15× speedup over off-the-shelf TensorFlow on a Raspberry Pi 4. For Large Language Models, PockEngine enabled fine-tuning of LlamaV2-7B on an NVIDIA Jetson AGX Orin at 550 tokens/s, which is 7.9× faster than the PyTorch baseline.

Memory Efficiency

The engine demonstrated remarkable memory efficiency, with savings ranging from 5.6× to 21.3× during backpropagation. A notable example is MCUNet, where the memory footprint was reduced from 3.6MB to a mere 173KB, making training feasible on extremely constrained microcontrollers.

Accuracy and Convergence

Despite the use of sparse updates and heavy pruning, PockEngine maintains high model quality. Across most vision and NLP tasks (including the GLUE benchmark), the sparse update schemes remained within 1% accuracy of full fine-tuning. Strategic freezing—such as updating only the attention modules in Transformers or the first convolution in CNN blocks—proved highly effective for adaptation.

Discussion and Limitations

PockEngine demonstrates that inference-optimized libraries can be repurposed for training if the backward pass is represented as a static graph. However, some challenges remain. The current "optimal" sparse update schemes are determined via an offline evolutionary search based on a cost model; future work could focus on making this search dynamic. Additionally, while PockEngine is highly optimized for fine-tuning and transfer learning, training large models from scratch on the edge remains computationally prohibitive. Further integration with quantization-aware training (QAT) could also lower the resource requirements for DSP-heavy environments.

Conclusion

PockEngine provides a tiny, sparse, and efficient solution for the growing demand for on-device learning. By combining compilation-first autodiff with system-aware sparse backpropagation, it enables sophisticated models like LlamaV2 to be fine-tuned on edge hardware with unprecedented speed and memory efficiency. This work paves the way for truly personalized and private AI that learns directly in the user's pocket.

Summary updated

Abstract

On-device learning and efficient fine-tuning enable continuous and privacy-preserving customization (e.g., locally fine-tuning large language models on personalized data). However, existing training frameworks are designed for cloud servers with powerful accelerators (e.g., GPUs, TPUs) and lack the optimizations for learning on the edge, which faces challenges of resource limitations and edge hardware diversity. We introduce PockEngine: a tiny, sparse and efficient engine to enable fine-tuning on various edge devices. PockEngine supports sparse backpropagation: it prunes the backward graph and sparsely updates the model with measured memory saving and latency reduction while maintaining the model quality. Secondly, PockEngine is compilation first: the entire training graph (including forward, backward and optimization steps) is derived at compile-time, which reduces the runtime overhead and brings opportunities for graph transformations. PockEngine also integrates a rich set of training graph optimizations, thus can further accelerate the training cost, including operator reordering and backend switching. PockEngine supports diverse applications, frontends and hardware backends: it flexibly compiles and tunes models defined in PyTorch/TensorFlow/Jax and deploys binaries to mobile CPU/GPU/DSPs. We evaluated PockEngine on both vision models and large language models. PockEngine achieves up to 15 $\times$ speedup over off-the-shelf TensorFlow (Raspberry Pi), 5.6 $\times$ memory saving back-propagation (Jetson AGX Orin). Remarkably, PockEngine enables fine-tuning LLaMav2-7B on NVIDIA Jetson AGX Orin at 550 tokens/s, 7.9$\times$ faster than the PyTorch.
BibTeX
@inproceedings{zhu2023pockengine,
  author = {Ligeng Zhu and Lanxiang Hu and Ji Lin and Wei-Chen Wang and Wei-Ming Chen and Chuang Gan and Song Han},
  title = {{PockEngine: Sparse and Efficient Fine-tuning in a Pocket}},
  booktitle = {MICRO},
  year = {2023},
  doi = {10.1145/3613424.3614307},
}
Figure