Abstract
On-device learning and efficient fine-tuning are essential for continuous, privacy-preserving customization of large models, such as Large Language Models (LLMs), on personalized data. However, traditional training frameworks are optimized for cloud-scale accelerators and lack the necessary adaptations for resource-constrained edge hardware. We present PockEngine, a sparse and efficient engine designed specifically for fine-tuning on diverse edge devices. PockEngine utilizes sparse backpropagation to prune the backward graph, significantly reducing memory overhead and latency while maintaining model quality. By adopting a "compilation-first" approach, PockEngine derives the entire training graph at compile-time, eliminating runtime overhead and enabling advanced graph transformations. Our evaluation demonstrates that PockEngine achieves up to 15× speedup over TensorFlow on Raspberry Pi and enables LlamaV2-7B fine-tuning at 550 tokens/s on NVIDIA Jetson AGX Orin, outperforming PyTorch by 7.9×.
Introduction
The paradigm of deep learning is shifting toward personalization, where models must adapt to individual user data locally to ensure privacy and low latency. Despite this shift, model training remains a "cloud-first" activity. Standard frameworks like PyTorch and TensorFlow are designed for massive GPU clusters; they carry significant runtime overhead (often exceeding 300MB for the host language alone) and lack the flexibility to handle the extreme resource limitations and hardware diversity of the edge.
Edge devices, ranging from ARM-based microcontrollers to mobile DSPs and Apple M-series chips, face two primary hurdles: limited memory/compute power and a fragmented hardware ecosystem. While libraries like TF-Lite and TensorRT have addressed edge inference, PockEngine fills the critical gap for on-device training.
System Design and Methodology
PockEngine departs from the traditional "runtime-differentiation" model in favor of a specialized compilation-based workflow.
Sparse Backpropagation (Sparse-BP)
Memory is the primary bottleneck for on-device training, largely due to the need to store intermediate activations for gradient calculation. PockEngine introduces Sparse-BP, which prunes the backward computation graph. By supporting bias-only, layer-wise, and sub-layer (channel-level) updates, the engine can release intermediate activations much earlier than standard frameworks. This system-algorithm co-design ensures that the reduction in FLOPs translates directly into significant memory savings.
Compilation-First Autodiff
Unlike frameworks that compute gradients dynamically at runtime, PockEngine derives the entire training graph—including forward, backward, and optimization steps—at compile-time. This approach eliminates Python runtime overhead and allows the compiler to perform aggressive dead-code elimination and graph-level optimizations.
Training Graph Optimizations
PockEngine integrates several novel optimizations to further accelerate the training process:
- Operator Reordering: Gradients are applied to weights immediately during the backward pass. This prevents the system from holding multiple large gradient tensors in memory simultaneously, reducing peak memory usage.
- Backend Switching: PockEngine can wrap "inference-only" vendor libraries (such as Qualcomm’s SNPE or NVIDIA’s TensorRT). It treats backward passes as standard forward operations, allowing it to leverage highly optimized proprietary kernels for training.
- Winograd for Training: The engine applies Winograd convolution—typically an inference-only optimization—to the frozen layers of the training graph to accelerate the overall execution.
Experimental Evaluation
We evaluated PockEngine across a diverse set of tasks and hardware backends, including vision models, BERT-based NLP models, and Large Language Models.
Performance and Speed
PockEngine achieved a 15× speedup over off-the-shelf TensorFlow on a Raspberry Pi 4. For Large Language Models, PockEngine enabled fine-tuning of LlamaV2-7B on an NVIDIA Jetson AGX Orin at 550 tokens/s, which is 7.9× faster than the PyTorch baseline.
Memory Efficiency
The engine demonstrated remarkable memory efficiency, with savings ranging from 5.6× to 21.3× during backpropagation. A notable example is MCUNet, where the memory footprint was reduced from 3.6MB to a mere 173KB, making training feasible on extremely constrained microcontrollers.
Accuracy and Convergence
Despite the use of sparse updates and heavy pruning, PockEngine maintains high model quality. Across most vision and NLP tasks (including the GLUE benchmark), the sparse update schemes remained within 1% accuracy of full fine-tuning. Strategic freezing—such as updating only the attention modules in Transformers or the first convolution in CNN blocks—proved highly effective for adaptation.
Discussion and Limitations
PockEngine demonstrates that inference-optimized libraries can be repurposed for training if the backward pass is represented as a static graph. However, some challenges remain. The current "optimal" sparse update schemes are determined via an offline evolutionary search based on a cost model; future work could focus on making this search dynamic. Additionally, while PockEngine is highly optimized for fine-tuning and transfer learning, training large models from scratch on the edge remains computationally prohibitive. Further integration with quantization-aware training (QAT) could also lower the resource requirements for DSP-heavy environments.
Conclusion
PockEngine provides a tiny, sparse, and efficient solution for the growing demand for on-device learning. By combining compilation-first autodiff with system-aware sparse backpropagation, it enables sophisticated models like LlamaV2 to be fine-tuned on edge hardware with unprecedented speed and memory efficiency. This work paves the way for truly personalized and private AI that learns directly in the user's pocket.
Summary updated