Designing Large Foundation Models for Efficient Training and Inference: A Survey

Dong Liu, Yanxuan Yu, Yite Wang, Jing Wu, Zhongwei Wan, Sina Alinejad, Benjamin Lengerich, Ying Nian WuarXiv 2024 · arXiv

TL;DR

This survey paper provides a comprehensive taxonomy of modern techniques designed to optimize the training and inference of large foundation models. By categorizing recent advancements into model design, system design, and model-system co-design, the authors map out how the AI community is making massive models computationally efficient, affordable, and deployable on resource-constrained hardware.

What Problem Does This Paper Address?

State-of-the-art foundation models—such as GPT-4 or large multimodal architectures—are notoriously resource-hungry. Training them can take several months, require thousands of interconnected GPUs, and cost millions of dollars. This creates a steep barrier to entry for most researchers and organizations. Furthermore, deploying these multi-billion-parameter models for inference on personal computers or mobile devices is practically impossible without severe optimization.

To tackle these challenges, researchers have developed a dizzying array of efficiency techniques scattered across different sub-fields. This paper addresses the lack of a unified framework by synthesizing these diverse methods into a structured overview, helping engineers and researchers navigate the trade-offs between computational cost and model performance.

Methods

The authors organize the landscape of efficiency technologies into three core paradigms:

  1. Model Design: Algorithmic modifications to the model's internal structure or numerical representations.
  2. Quantization: Lowering the precision of weights and activations (e.g., converting FP32 to INT8, FP8, or even extreme 1-bit/2-bit formats). This includes Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).
  3. Knowledge Distillation (KD): Compressing a large "teacher" model into a smaller "student" model using either soft probability distributions or hard labels.
  4. Pruning: Removing redundant weights or structural components (unstructured vs. structured pruning) based on saliency metrics.
  5. System Design: Infrastructure and hardware-level optimizations that do not alter the model's fundamental physics but change how it executes. This includes Key-Value (KV) cache compression, token reduction/merging, memory paging (like vLLM), and low-level GPU kernel optimizations.
  6. Model-System Co-Design: Bridging the gap between software algorithms and hardware execution. Prominent examples include Mixture of Experts (MoE) for dynamic routing, Mixed-Precision Training (MPT), and Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA combined with system-level quantization.

An overview of this taxonomy is illustrated below:

Figure 1: Efficient Foundation Models Overview

Datasets

As a survey paper, this work does not introduce any new datasets or benchmarks. Instead, it reviews literature that evaluates efficiency methods across standard foundation model families (such as LLaMA, OPT, and Baichuan) and various vision-language datasets.

Key Insights and Ideas

  • The Rise of FP8: While INT8 was previously the standard for quantization, newer formats like FP8 (E5M2 and E4M3) are gaining traction. FP8 offers a wider dynamic range, making it significantly better at handling the large activation outliers common in modern Large Language Models (LLMs).
  • Exploiting Natural Sparsity: The survey highlights that ReLU-based Transformers naturally exhibit massive activation sparsity. In models like OPT-175B, over 95% of the feedforward network (FFN) activations are zero, presenting a massive, underutilized opportunity for inference acceleration.
  • The KV Cache Bottleneck: During autoregressive generation, memory bandwidth—not raw compute—is often the primary bottleneck due to the growing size of the KV cache. This has driven a massive wave of research into adaptive precision cache eviction and low-rank approximations.
  • Hardware-Aware Gating: In sparse MoE systems, a major system challenge is load balancing (ensuring tokens are distributed evenly across experts). The paper emphasizes that algorithmic fixes (like auxiliary losses) must be paired with topology-aware routing to minimize expensive cross-node communication.

Key Results

The paper compiles key quantitative milestones from the surveyed literature:

  • Quantization Feasibility: The authors map out how various methods successfully compress LLMs and Diffusion models down to 4-bit or 2-bit precision with minimal accuracy degradation.
Quantization Methods for Foundation Models
  • Pruning Efficiency: Structured pruning methods like LLMPruner can achieve a 20% parameter reduction while retaining 90% of the original performance, drastically accelerating inference.
  • Token and Cache Optimization: Systems like VL-Cache achieve up to a 90% memory reduction in vision-language tasks alongside a 7× improvement in decoding latency.
  • Hardware Alignment: The survey provides a handy reference mapping out native precision support across modern NVIDIA GPU architectures, highlighting how newer chips enable advanced mixed-precision training.
NVIDIA GPU Precision Support Summary

Limitations and Future Work

The authors identify several open challenges for the community:

  • Tighter Hardware-Software Integration: Future work needs to move toward a tighter coupling of sparse fine-tuning algorithms with custom GPU hardware accelerators (e.g., leveraging native tensor sparsity).
  • MoE Out-of-Distribution Vulnerabilities: Sparse MoE models still struggle with load balancing when faced with out-of-distribution data or very small inference batch sizes, necessitating more robust routing mechanisms like "Soft MoE."
  • Scaling to Extreme Contexts: Even with alternative architectures like State-Space Models (e.g., Mamba) and linear attention, handling extremely long contexts (128K+ tokens) efficiently at the system level remains an open problem.

Summary updated

Abstract

This paper focuses on modern efficient training and inference technologies on foundation models and illustrates them from two perspectives: model and system design. Model and System Design optimize LLM training and inference from different aspects to save computational resources, making LLMs more efficient, affordable, and more accessible. The paper list repository is available at https://github.com/NoakLiu/Efficient-Foundation-Models-Survey.
BibTeX
@misc{liu2024designing,
  author = {Dong Liu and Yanxuan Yu and Yite Wang and Jing Wu and Zhongwei Wan and Sina Alinejad and Benjamin Lengerich and Ying Nian Wu},
  title = {{Designing Large Foundation Models for Efficient Training and Inference: A Survey}},
  year = {2024},
  eprint = {2409.01990},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.01990},
}

Loading related content...

Figure