Introduction
Serving modern Large Language Models (LLMs) like GPT-3 or Llama-2 has become a significant economic challenge. Traditional hardware architectures, including GPUs and TPUs, are encountering a "scalability wall." This bottleneck is primarily due to the memory bandwidth requirements of LLM inference; compute units often remain idle while waiting for data from external High Bandwidth Memory (HBM).
To address this, the "Chiplet Cloud" architecture proposes a shift toward specialized ASICs that prioritize Total Cost of Ownership (TCO) per token by moving model parameters onto a scalable, on-chip SRAM system.
Architectural Methodology
The researchers propose a three-pronged solution to optimize LLM serving:
- Parameters-on-Chip (CC-MEM): By utilizing a scalable on-chip SRAM system instead of external DRAM, the architecture eliminates the bandwidth bottleneck.
- Chiplet-Based Design: Using small, replicated chiplets improves manufacturing yields and allows for modular scaling.
- Store-as-Compressed, Load-as-Dense: Integrated hardware decompression allows the system to benefit from model sparsity without requiring complex sparse-compute units.
A two-phase co-design methodology was employed to balance physical hardware constraints with software mapping strategies, ultimately identifying a "Goldilocks" chiplet size of approximately 200 mm².
Key Results and Performance
The Chiplet Cloud architecture demonstrates significant improvements over existing cloud infrastructure:
- TCO Efficiency: Achieved a 97× improvement in TCO per token compared to rented A100 GPU clouds.
- Fabrication Gains: Even when comparing owned hardware, Chiplet Cloud outperformed GPUs by 8.3× and TPUs by 3.7×.
- Sparsity Support: The system can accommodate 1.7× larger models by leveraging 60% unstructured sparsity, further reducing TCO by 7.4%.
Conclusion
While the architecture is highly optimized for Transformer-based decoders, its massive TCO advantages suggest that specialized ASICs are the key to democratizing large-scale AI. Future work will explore 3D packaging and tighter software-hardware integration to further push the boundaries of LLM serving efficiency.
Summary updated