Abstract
Big pre-trained models (PTMs) have received increasing attention in recent years from academia and industry for their excellent performance on downstream tasks. However, huge computing power and sophisticated technical expertise are required to develop big models, discouraging many institutes and researchers. In order to facilitate the popularization of big models, we introduce OpenBMB, an open-source suite of big models, to break the barriers of computation and expertise of big model applications. In this chapter, we will introduce the core toolkits in OpenBMB, including BMTrain for efficient training, OpenPrompt and OpenDelta for efficient tuning, BMCook for efficient compression, and BMInf for efficient inference.
Introduction
The "Big Model" era, sparked by the success of architectures like BERT and GPT-3, has fundamentally shifted the landscape of AI research. However, this progress has created a significant divide. Developing and deploying these models faces two primary hurdles: the Computation Barrier, where training and inference require prohibitively expensive high-performance hardware, and the Expertise Barrier, where standard deep learning techniques fail to scale efficiently. OpenBMB addresses these gaps by providing a "full-stack" system that simplifies the lifecycle of a big model, from initial training to deployment on consumer-grade hardware.
Core Toolkits and Methodology
The OpenBMB ecosystem is built around five core toolkits, each targeting a specific stage of the model pipeline:
BMTrain: Efficient Large-Scale Training
BMTrain implements advanced distributed strategies like the Zero Redundancy Optimizer (ZeRO). While standard data parallelism replicates the entire model on every GPU, ZeRO partitions parameters, gradients, and optimizer states across the cluster. This approach drastically reduces the memory footprint per device, enabling the training of models that would otherwise exceed single-GPU capacity.
OpenPrompt & OpenDelta: Parameter-Efficient Tuning
These tools focus on "Parameter-Efficient Fine-Tuning" (PEFT). Instead of updating billions of parameters, they utilize Prompt Learning (OpenPrompt), which adapts models using task instructions in natural language, and Delta Tuning (OpenDelta), which injects small, trainable modules like LoRA or Adapters into a frozen backbone model.
BMCook & BMInf: Compression and Inference
BMCook provides a unified framework for model compression, combining quantization, distillation, and pruning. Notably, it supports hardware-aware 2:4 sparsity to leverage NVIDIA's Sparse Tensor Cores. BMInf enables inference on consumer hardware through kernel fusion and a sophisticated CPU-GPU scheduling mechanism. By dynamically "swapping" model parameters between CPU RAM and GPU VRAM, it allows 10B+ parameter models to run on GPUs with limited memory.
Key Results and Impact
OpenBMB demonstrates significant improvements in accessibility and performance:
- Training Scale: BMTrain enables training a 175-billion parameter model on 64x A100 GPUs, a feat typically requiring far more resources with standard data parallelism.
- Inference Accessibility: Using BMInf, a 10-billion parameter model can run on a consumer-grade NVIDIA GTX 1060 (6GB VRAM), achieving a decoding speed of 20 tokens/s—a nearly 7x improvement over standard PyTorch implementations.
Conclusion
OpenBMB democratizes large-scale AI by treating memory as a tiered resource and providing optimized, easy-to-use toolkits. By breaking the computation and expertise barriers, it empowers a broader range of researchers to participate in the development and application of big pre-trained models.
Summary updated