Hermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices

Xueyuan Han, Zinuo Cai, Yichu Zhang, Chongxin Fan, Junhan Liu, Ruhui Ma, Rajkumar Buyya

Abstract

Large Language Models (LLMs) and Vision Transformers (ViTs) are increasingly difficult to deploy on resource-constrained edge devices due to their massive memory footprints. This paper presents Hermes, a novel framework designed to execute large Transformer models on edge hardware without sacrificing accuracy or requiring extensive model compression. By implementing a specialized pipeline mechanism called PIPELOAD, Hermes achieves up to a 4.24× speedup and a 90.3% reduction in memory usage compared to existing state-of-the-art methods.

Introduction

The rapid advancement of AI has led to models with billions of parameters, such as GPT-3 and ViT-Large. While these models offer transformative capabilities, they face a significant "memory wall" on edge devices like the Raspberry Pi or NVIDIA Jetson Nano, which typically possess only 4GB to 8GB of RAM. Current solutions often force a trade-off: either reduce accuracy through pruning and quantization or suffer high latency through standard memory swapping. Standard swapping is particularly inefficient because disk I/O is significantly slower than CPU computation, leading to massive pipeline stalls.

The Loading Bottleneck

A critical observation in this research is the disparity between loading and computation times. For most Transformer-based models, loading a layer from disk into memory takes approximately 10 times longer than the actual inference computation.

Latency Breakdown of Transformer Models

As shown in the latency analysis, load latency accounts for over 90% of the total execution time for models like BERT-Large and GPT-2. This bottleneck is the primary cause of performance degradation in naive memory-swapping implementations.

System Architecture: PIPELOAD

Hermes addresses the loading bottleneck through PIPELOAD, a memory-efficient pipeline execution mechanism that overlaps disk I/O with computation. The system orchestrates three specialized types of agents to manage model layers dynamically.

Hermes System Architecture and Workflow

Core Components

  1. Loading Agents: Multiple parallel workers that pre-fetch model layers from disk into memory. By employing multiple agents, the system can mask the slow disk-loading time by starting the transfer of future layers while the current layer is being processed.
  2. Inference Agent: A single worker dedicated to performing mathematical computations on the CPU. It pulls ready layers from an inference queue to ensure sequential execution.
  3. Daemon Agent: This component manages the memory lifecycle. It monitors usage and issues "destruction signals" to immediately purge model weights from RAM once their inference task is complete, maintaining a minimal memory footprint.

Evaluation and Results

The researchers evaluated Hermes across four representative models: BERT-Large, ViT-Large, GPT-2, and GPT-J. The framework demonstrated significant improvements over PipeSwitch, the current state-of-the-art baseline:

  • Inference Speed: Hermes achieved up to a 4.24× increase in speed for BERT and ViT models, and a 2.58× increase for GPT-style models.
  • Memory Efficiency: The framework reduced memory consumption by up to 90.3%. In extreme scenarios, Hermes successfully ran a ViT model with only 60MB of available RAM, a condition that would cause standard methods to crash.
  • Flexibility: Unlike static compression, Hermes dynamically adjusts the number of Loading Agents based on real-time memory availability, making it ideal for multi-tasking edge environments.

Conclusion

Hermes effectively bridges the gap between massive Transformer models and memory-constrained edge hardware. By focusing on the dynamic loading and destruction of heavy encoder/decoder layers and masking I/O latency through parallel loading agents, Hermes provides a high-performance, accuracy-preserving solution for edge AI. Future work will focus on extending this framework to mobile GPUs and optimizing the repetitive loading patterns required for autoregressive text generation.

Summary updated

BibTeX
@inproceedings{han2024hermes,
  author = {Xueyuan Han and Zinuo Cai and Yichu Zhang and Chongxin Fan and Junhan Liu and Ruhui Ma and Rajkumar Buyya},
  title = {{Hermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices}},
  booktitle = {ICCD},
  year = {2024},
  doi = {10.1109/iccd63220.2024.00076},
}
Figure