Who Says Elephants Can’t Run: Bringing Large Scale MoE Models into Cloud Scale Production

Young Jin Kim, Rawn Henry, Raffy Fahim, Hany Hassan2022 · DOI

Abstract

Mixture of Experts (MoE) models with conditional execution of sparsely activated layers has enabled training models with a much larger number of parameters. As a result, these models have achieved significantly better quality on various natural language processing tasks including machine translation. However, it remains challenging to deploy such models in real-life scenarios due to the large memory requirements and inefficient inference. In this work, we introduce a highly efficient inference framework with several optimization approaches to accelerate the computation of sparse models and cut down the memory consumption significantly. While we achieve up to 26x speed-up in terms of throughput, we also reduce the model size almost to one eighth of the original 32-bit float model by quantizing expert weights into 4-bit integers. As a result, we are able to deploy 136x larger models with 27% less cost and significantly better quality with large scale MoE model deployment compared to the existing solutions. This enables a paradigm shift in deploying large scale multilingual MoE transformers models instead of distilling into dozens of smaller models per language or task.

Introduction

Large-scale Transformer models continue to push the boundaries of natural language processing, but their deployment costs have scaled alongside their performance. Mixture of Experts (MoE) architectures offer a potential solution by allowing models to scale to hundreds of billions of parameters while only activating a small fraction for any given input. However, the practical deployment of these "elephant-sized" models is often hindered by massive memory footprints and high inference latency. Naive implementations can be up to 30x slower than dense counterparts due to routing logic and communication overhead.

Traditionally, industry practitioners have bypassed these issues by distilling massive MoE models into numerous smaller, language-specific models for CPU deployment. This work proposes an alternative: a highly optimized inference framework that allows a single, massive multilingual MoE model to run directly on GPUs with superior efficiency and lower cost.

Methodology: Optimized Inference Framework

To bridge the gap between MoE potential and production reality, we built an inference framework on top of NVIDIA’s FasterTransformer, introducing several key optimizations:

Optimized Routing and Grouped GEMM

Token routing is treated as a parallel sorting and grouping problem. Using the CUB library's radix sort, tokens are grouped by their assigned experts. We then utilize Grouped GEMM from the CUTLASS library to process all experts in parallel within a single kernel. This approach avoids the latency penalty of launching separate tasks for each expert and significantly streamlines the computation path.

Figure 1: Illustration of the Grouped GEMM approach for processing MoE experts in parallel.

Weight-Only Quantization

To fit large models onto standard production GPUs (e.g., 16GB T4), we implemented 4-bit and 8-bit weight-only quantization. While weights are quantized to reduce memory, activations are maintained in FP16 to preserve precision. We developed custom fused GEMM kernels that dequantize weights on-the-fly, eliminating the need to store intermediate dequantized values in memory.

Dynamic Batch Pruning

In sequence generation tasks like machine translation, different sentences in a batch finish at different times. Our framework implements batch pruning, which dynamically removes finished sequences from MoE layers. This prevents the GPU from wasting cycles loading expert weights for tokens that have already been fully processed.

Experimental Results

We evaluated the framework using a production-scale multilingual machine translation system trained on 4 billion sentence pairs across 10 language pairs.

Throughput and Cost Efficiency

The framework achieved up to a 26x throughput speedup compared to standard PyTorch implementations. More importantly, it enabled the deployment of a 5.32 billion parameter MoE model on a single NVIDIA T4 GPU. As shown in the comparison below, this configuration allows for a model 136x larger than traditional distilled CPU models while reducing the cost per token by 27%.

Figure 4: Comparison of hardware performance and cost, showing the efficiency of MoE deployment on NVIDIA T4 GPUs.

Quality Preservation

A critical finding was that weight-only quantization resulted in negligible quality loss. The average BLEU score drop across 10 language pairs was only -0.007 for 8-bit and -0.167 for 4-bit quantization. This trade-off is highly favorable given the 8x reduction in model memory requirements.

Conclusion

By optimizing routing logic and employing aggressive weight-only quantization, we have demonstrated that large-scale MoE models can be deployed efficiently in cloud-scale production. This approach eliminates the need for complex distillation pipelines and allows a single, high-quality multilingual model to serve diverse tasks at a lower cost than traditional specialized models. Future work will explore activation quantization and further optimization for models spanning multiple GPUs.

Summary updated

Abstract

Mixture of Experts (MoE) models with conditional execution of sparsely activated layers has enabled training models with a much larger number of parameters. As a result, these models have achieved significantly better quality on various natural language processing tasks including machine translation. However, it remains challenging to deploy such models in real-life scenarios due to the large memory requirements and inefficient inference. In this work, we introduce a highly efficient inference framework with several optimization approaches to accelerate the computation of sparse models and cut down the memory consumption significantly. While we achieve up to 26x speed-up in terms of throughput, we also reduce the model size almost to one eighth of the original 32-bit float model by quantizing expert weights into 4-bit integers. As a result, we are able to deploy 136x larger models with 27% less cost and significantly better quality with large scale MoE model deployment compared to the existing solutions. This enables a paradigm shift in deploying large scale multilingual MoE transformers models instead of distilling into dozens of smaller models per language or task.
BibTeX
@misc{kim2022who,
  author = {Young Jin Kim and Rawn Henry and Raffy Fahim and Hany Hassan},
  title = {{Who Says Elephants Can’t Run: Bringing Large Scale MoE Models into Cloud Scale Production}},
  year = {2022},
  doi = {10.18653/v1/2022.sustainlp-1.6},
}
Figure