Chiplet Cloud: Building AI Supercomputers for Serving Large Generative Language Models

Huwan Peng, Scott Davidson, Richard Shi, Shuaiwen Leon Song, Michael B. TaylorarXiv 2023 · DOI · arXiv

Introduction

Serving modern Large Language Models (LLMs) like GPT-3 or Llama-2 has become a significant economic challenge. Traditional hardware architectures, including GPUs and TPUs, are encountering a "scalability wall." This bottleneck is primarily due to the memory bandwidth requirements of LLM inference; compute units often remain idle while waiting for data from external High Bandwidth Memory (HBM).

Transformer architecture and computational intensity of LLM serving.

To address this, the "Chiplet Cloud" architecture proposes a shift toward specialized ASICs that prioritize Total Cost of Ownership (TCO) per token by moving model parameters onto a scalable, on-chip SRAM system.

Architectural Methodology

The researchers propose a three-pronged solution to optimize LLM serving:

  1. Parameters-on-Chip (CC-MEM): By utilizing a scalable on-chip SRAM system instead of external DRAM, the architecture eliminates the bandwidth bottleneck.
  2. Chiplet-Based Design: Using small, replicated chiplets improves manufacturing yields and allows for modular scaling.
  3. Store-as-Compressed, Load-as-Dense: Integrated hardware decompression allows the system to benefit from model sparsity without requiring complex sparse-compute units.
The three core optimization strategies of the Chiplet Cloud architecture.

A two-phase co-design methodology was employed to balance physical hardware constraints with software mapping strategies, ultimately identifying a "Goldilocks" chiplet size of approximately 200 mm².

Key Results and Performance

The Chiplet Cloud architecture demonstrates significant improvements over existing cloud infrastructure:

  • TCO Efficiency: Achieved a 97× improvement in TCO per token compared to rented A100 GPU clouds.
  • Fabrication Gains: Even when comparing owned hardware, Chiplet Cloud outperformed GPUs by 8.3× and TPUs by 3.7×.
  • Sparsity Support: The system can accommodate 1.7× larger models by leveraging 60% unstructured sparsity, further reducing TCO by 7.4%.

Conclusion

While the architecture is highly optimized for Transformer-based decoders, its massive TCO advantages suggest that specialized ASICs are the key to democratizing large-scale AI. Future work will explore 3D packaging and tighter software-hardware integration to further push the boundaries of LLM serving efficiency.

Summary updated

BibTeX
@misc{peng2023chiplet,
  author = {Huwan Peng and Scott Davidson and Richard Shi and Shuaiwen Leon Song and Michael B. Taylor},
  title = {{Chiplet Cloud: Building AI Supercomputers for Serving Large Generative Language Models}},
  year = {2023},
  eprint = {2307.02666},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.02666},
}
Figure