OpenBMB: Big Model Systems for Large-Scale Representation Learning

Guoyang Zeng, Xu Han, Zhengyan Zhang, Zhiyuan Liu, Yankai Lin, Maosong SunOpenalex Unknown 2023 · DOI · OpenAlex

Abstract

Big pre-trained models (PTMs) have received increasing attention in recent years from academia and industry for their excellent performance on downstream tasks. However, huge computing power and sophisticated technical expertise are required to develop big models, discouraging many institutes and researchers. In order to facilitate the popularization of big models, we introduce OpenBMB, an open-source suite of big models, to break the barriers of computation and expertise of big model applications. In this chapter, we will introduce the core toolkits in OpenBMB, including BMTrain for efficient training, OpenPrompt and OpenDelta for efficient tuning, BMCook for efficient compression, and BMInf for efficient inference.

Introduction

The "Big Model" era, sparked by the success of architectures like BERT and GPT-3, has fundamentally shifted the landscape of AI research. However, this progress has created a significant divide. Developing and deploying these models faces two primary hurdles: the Computation Barrier, where training and inference require prohibitively expensive high-performance hardware, and the Expertise Barrier, where standard deep learning techniques fail to scale efficiently. OpenBMB addresses these gaps by providing a "full-stack" system that simplifies the lifecycle of a big model, from initial training to deployment on consumer-grade hardware.

An overview of the OpenBMB architecture, illustrating the integrated pipeline from training and tuning to model compression and inference.

Core Toolkits and Methodology

The OpenBMB ecosystem is built around five core toolkits, each targeting a specific stage of the model pipeline:

BMTrain: Efficient Large-Scale Training

BMTrain implements advanced distributed strategies like the Zero Redundancy Optimizer (ZeRO). While standard data parallelism replicates the entire model on every GPU, ZeRO partitions parameters, gradients, and optimizer states across the cluster. This approach drastically reduces the memory footprint per device, enabling the training of models that would otherwise exceed single-GPU capacity.

ZeRO partitions model states across GPUs, utilizing ReduceScatter and AllGather operations to minimize redundancy and enable the training of massive models.

OpenPrompt & OpenDelta: Parameter-Efficient Tuning

These tools focus on "Parameter-Efficient Fine-Tuning" (PEFT). Instead of updating billions of parameters, they utilize Prompt Learning (OpenPrompt), which adapts models using task instructions in natural language, and Delta Tuning (OpenDelta), which injects small, trainable modules like LoRA or Adapters into a frozen backbone model.

BMCook & BMInf: Compression and Inference

BMCook provides a unified framework for model compression, combining quantization, distillation, and pruning. Notably, it supports hardware-aware 2:4 sparsity to leverage NVIDIA's Sparse Tensor Cores. BMInf enables inference on consumer hardware through kernel fusion and a sophisticated CPU-GPU scheduling mechanism. By dynamically "swapping" model parameters between CPU RAM and GPU VRAM, it allows 10B+ parameter models to run on GPUs with limited memory.

Key Results and Impact

OpenBMB demonstrates significant improvements in accessibility and performance:

  • Training Scale: BMTrain enables training a 175-billion parameter model on 64x A100 GPUs, a feat typically requiring far more resources with standard data parallelism.
  • Inference Accessibility: Using BMInf, a 10-billion parameter model can run on a consumer-grade NVIDIA GTX 1060 (6GB VRAM), achieving a decoding speed of 20 tokens/s—a nearly 7x improvement over standard PyTorch implementations.

Conclusion

OpenBMB democratizes large-scale AI by treating memory as a tiered resource and providing optimized, easy-to-use toolkits. By breaking the computation and expertise barriers, it empowers a broader range of researchers to participate in the development and application of big pre-trained models.

Summary updated

Abstract

Abstract Big pre-trained models (PTMs) have received increasing attention in recent years from academia and industry for their excellent performance on downstream tasks. However, huge computing power and sophisticated technical expertise are required to develop big models, discouraging many institutes and researchers. In order to facilitate the popularization of big models, we introduce OpenBMB, an open-source suite of big models, to break the barriers of computation and expertise of big model applications. In this chapter, we will introduce the core toolkits in OpenBMB, including BMTrain for efficient training, OpenPrompt and OpenDelta for efficient tuning, BMCook for efficient compression, and BMInf for efficient inference.
BibTeX
@inproceedings{zeng2023openbmb,
  author = {Guoyang Zeng and Xu Han and Zhengyan Zhang and Zhiyuan Liu and Yankai Lin and Maosong Sun},
  title = {{OpenBMB: Big Model Systems for Large-Scale Representation Learning}},
  booktitle = {Openalex Unknown},
  year = {2023},
  doi = {10.1007/978-981-99-1600-9\_13},
}
Figure