Abstract
Large Language Models (LLMs) and Vision Transformers (ViTs) are increasingly difficult to deploy on resource-constrained edge devices due to their massive memory footprints. This paper presents Hermes, a novel framework designed to execute large Transformer models on edge hardware without sacrificing accuracy or requiring extensive model compression. By implementing a specialized pipeline mechanism called PIPELOAD, Hermes achieves up to a 4.24× speedup and a 90.3% reduction in memory usage compared to existing state-of-the-art methods.
Introduction
The rapid advancement of AI has led to models with billions of parameters, such as GPT-3 and ViT-Large. While these models offer transformative capabilities, they face a significant "memory wall" on edge devices like the Raspberry Pi or NVIDIA Jetson Nano, which typically possess only 4GB to 8GB of RAM. Current solutions often force a trade-off: either reduce accuracy through pruning and quantization or suffer high latency through standard memory swapping. Standard swapping is particularly inefficient because disk I/O is significantly slower than CPU computation, leading to massive pipeline stalls.
The Loading Bottleneck
A critical observation in this research is the disparity between loading and computation times. For most Transformer-based models, loading a layer from disk into memory takes approximately 10 times longer than the actual inference computation.
As shown in the latency analysis, load latency accounts for over 90% of the total execution time for models like BERT-Large and GPT-2. This bottleneck is the primary cause of performance degradation in naive memory-swapping implementations.
System Architecture: PIPELOAD
Hermes addresses the loading bottleneck through PIPELOAD, a memory-efficient pipeline execution mechanism that overlaps disk I/O with computation. The system orchestrates three specialized types of agents to manage model layers dynamically.
Core Components
- Loading Agents: Multiple parallel workers that pre-fetch model layers from disk into memory. By employing multiple agents, the system can mask the slow disk-loading time by starting the transfer of future layers while the current layer is being processed.
- Inference Agent: A single worker dedicated to performing mathematical computations on the CPU. It pulls ready layers from an inference queue to ensure sequential execution.
- Daemon Agent: This component manages the memory lifecycle. It monitors usage and issues "destruction signals" to immediately purge model weights from RAM once their inference task is complete, maintaining a minimal memory footprint.
Evaluation and Results
The researchers evaluated Hermes across four representative models: BERT-Large, ViT-Large, GPT-2, and GPT-J. The framework demonstrated significant improvements over PipeSwitch, the current state-of-the-art baseline:
- Inference Speed: Hermes achieved up to a 4.24× increase in speed for BERT and ViT models, and a 2.58× increase for GPT-style models.
- Memory Efficiency: The framework reduced memory consumption by up to 90.3%. In extreme scenarios, Hermes successfully ran a ViT model with only 60MB of available RAM, a condition that would cause standard methods to crash.
- Flexibility: Unlike static compression, Hermes dynamically adjusts the number of Loading Agents based on real-time memory availability, making it ideal for multi-tasking edge environments.
Conclusion
Hermes effectively bridges the gap between massive Transformer models and memory-constrained edge hardware. By focusing on the dynamic loading and destruction of heavy encoder/decoder layers and masking I/O latency through parallel loading agents, Hermes provides a high-performance, accuracy-preserving solution for edge AI. Future work will focus on extending this framework to mobile GPUs and optimizing the repetitive loading patterns required for autoregressive text generation.
Summary updated