Elixir: Train a Large Language Model on a Small GPU Cluster

Haichen Huang, Jiarui Fang, Hongxin Liu, Shenggui Li, Yang YouarXiv 2022 · DOI · arXiv

Abstract

Elixir is an automated system designed to facilitate the training of Large Language Models (LLMs) on resource-constrained GPU clusters. By integrating a pre-runtime profiler with a novel memory management layer called rCache, Elixir identifies the optimal balance between memory partitioning and offloading. This approach achieves up to a 3.4x speedup over state-of-the-art solutions like DeepSpeed, effectively lowering the hardware barrier for training massive models.

Introduction

The rapid advancement of Large Language Models has created a significant hardware barrier. While models like GPT-3 or PaLM feature hundreds of billions of parameters, individual GPU memory has plateaued around 80GB. To bridge this gap, researchers typically rely on memory partitioning (e.g., ZeRO) or memory offloading to CPU/NVMe storage.

Comparison of memory management strategies between standard PyTorch, ZeRO variants, and Elixir.

However, these techniques are notoriously difficult to tune. For example, DeepSpeed’s ZeRO-3 requires navigating over 18 adjustable parameters. Suboptimal configurations lead to either "Out of Memory" (OOM) errors or significant performance degradation. Elixir automates this process, making efficient LLM training accessible to researchers without extensive distributed systems expertise.

System Architecture and Methods

Elixir introduces a streamlined workflow to optimize training performance through four core components:

1. Pre-runtime Profiler

Before training commences, Elixir scans the model to construct a computation graph and measure memory requirements. This profiler is highly efficient, capable of analyzing a 175B parameter model on a single A100 GPU in approximately 10 seconds.

2. Parameter Chunking

To simplify memory management, Elixir flattens and concatenates individual parameters into uniform, one-dimensional "chunks." This abstraction ensures that communication operations fully utilize available hardware bandwidth by avoiding the overhead of managing thousands of small, irregularly sized tensors.

3. rCache (Redundancy Cache)

rCache is Elixir's primary innovation. It serves as a dedicated memory layer in GPU DRAM that stores gathered chunks. While ZeRO-3 partitions all states and ZeRO-2 replicates them, rCache provides fine-grained control, deciding exactly which chunks to retain on the GPU to maximize throughput while staying within strict memory limits.

The Elixir system workflow, featuring the profiler, search engine, and model wrapper.

4. Automated Search Engine

Using data from the profiler, Elixir runs a simulation to determine the optimal configuration. It calculates the "normalized benefit" of caching specific chunks versus offloading them, ensuring the hardware is utilized to its absolute limit without crashing.

Evaluation and Key Results

The system was evaluated using GPT-2 architecture models ranging from 4B to 20B parameters. Elixir was compared against industry standards including Megatron-LM, PyTorch FSDP, and DeepSpeed (ZeRO-2/3).

  • Performance Gains: Elixir achieved up to a 3.4x speedup on A100 80GB GPUs and a 3x speedup on A100 40GB GPUs compared to the best-tuned DeepSpeed configurations.
  • Robustness: In scenarios where Megatron-LM and FSDP failed due to OOM errors (particularly when training 15B+ models on limited GPUs), Elixir maintained stable and efficient training.
  • Scalability: As model size increased, Elixir’s performance remained more consistent than DeepSpeed, which often suffers from throughput drops when forced into aggressive offloading strategies.

Discussion and Future Work

Elixir demonstrates that the "middle ground" between total partitioning and total offloading is often the most efficient. By mathematically analyzing the benefits of caching versus offloading, the system outperforms manual human tuning.

Currently, Elixir is optimized for small clusters (1–4 GPUs). Future work will focus on integrating more complex "3D parallelism" (Tensor, Pipeline, and Data parallelism) to support massive-scale training across thousands of GPUs. Elixir is currently available as a beta component within the Colossal-AI ecosystem.

Summary updated

Abstract

In recent years, large language models have achieved great success due to their unprecedented size. However, training these models poses a challenge for most researchers as it requires a substantial number of GPUs. To reduce GPU memory usage, memory partitioning, and memory offloading have been proposed. These approaches eliminate memory redundancies and offload memory usage to the CPU and NVMe memory, respectively, enabling training on small GPU clusters. However, directly deploying these solutions often leads to suboptimal efficiency. Only experienced experts can unleash the full potential of hardware by carefully tuning the distributed configuration. Thus, we present a novel solution, Elixir, which automates efficient large-model training based on pre-runtime model profiling. Elixir aims to identify the optimal combination of partitioning and offloading techniques to maximize training throughput. In our experiments, Elixir significantly outperforms the current state-of-the-art baseline. Our optimal configuration achieves up to a 3.4$\times$ speedup on GPT-2 models compared with SOTA solutions. We hope that our work will benefit individuals who lack computing resources and expertise, granting them access to large models. The beta version of Elixir is now available at https://github.com/hpcaitech/ColossalAI/tree/feature/elixir.
BibTeX
@misc{huang2022elixir,
  author = {Haichen Huang and Jiarui Fang and Hongxin Liu and Shenggui Li and Yang You},
  title = {{Elixir: Train a Large Language Model on a Small GPU Cluster}},
  year = {2022},
  eprint = {2212.05339},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.05339},
}
Figure