Survey and Evaluation of Converging Architecture in LLMs based on Footsteps of Operations

Seongho Kim, Jihyun Moon, Juntaek Oh, Insu Choi, Joon-Sung Yang

Abstract

The advent of the Attention mechanism and Transformer architecture enables contextually natural text generation and compresses the burden of processing entire source information into singular vectors. Based on these two main ideas, model sizes gradually increases to accommodate more precise and comprehensive information, leading to the current state-of-the-art LLMs being very large, with parameters around 70 billion. As the model sizes are growing, the demand for substantial storage and computational capacity increases. This leads to the development of high-bandwidth memory and accelerators, as well as a variety of model architectures designed to meet these requirements. We note that LLM architectures have increasingly converged. This paper analyzes how these converged architectures perform in terms of layer configurations, operational mechanisms, and model sizes, considering various hyperparameter settings. In this paper, we conduct a concise survey of the history of LLMs by tracing the evolution of their operational improvements. Furthermore, we summarize the performance trends of LLMs under various hyperparameter settings using the RTX 6000, which features the state-of-the-art Ada Lovelace architecture. We conclude that even the same model can exhibit different behaviors depending on the hyperparameters or whether it is deployed in server or edge environments.

Introduction

As Large Language Models (LLMs) scale to tens of billions of parameters, their underlying architectures have begun to exhibit a remarkable degree of convergence. While early natural language processing (NLP) relied on sequential processing, modern state-of-the-art models like Llama 3 and Gemma have settled on a highly optimized, decoder-only Transformer design. This paper explores the "footsteps of operations" that led to this convergence and evaluates how these architectural choices interact with modern hardware, specifically analyzing the shifting bottlenecks between computation and memory access during inference.

The Evolution of LLM Architectures

The journey to modern LLMs began with Recurrent Neural Networks (RNNs) and LSTMs, which processed information sequentially. The introduction of the Encoder-decoder architecture and the subsequent Attention mechanism revolutionized the field by allowing models to focus on relevant parts of the input regardless of distance. The Transformer architecture further parallelized this process, leading to Pre-trained Language Models (PLMs) and eventually the massive decoder-only Large Language Models we see today.

The evolution of model architectures from early RNNs to modern converged LLM designs.

Methodology and Benchmarking

To understand the performance implications of these converged designs, we benchmarked four prominent open-source models: Llama2-7B, Llama3-8B, Gemma-2B, and Gemma-7B. The evaluation was conducted using NVIDIA’s TensorRT-LLM library on an RTX 6000 Ada GPU, utilizing the ccdv/cnn_dailymail dataset for text summarization tasks.

The analysis distinguishes between two critical phases of inference:

  1. Summarization Stage (Prefill): Processing the initial input prompt.
  2. Generation Stage (Decoding): Producing new tokens sequentially, one by one.

Operational Bottlenecks: Compute vs. Memory

A key finding of this study is the "Compute-Memory Flip" that occurs between inference stages.

  • Prefill is Compute-Bound: Because the entire input is processed simultaneously, this stage is dominated by General Matrix-Matrix multiplication (GEMM), which is highly parallelizable and utilizes the GPU's computational throughput.
  • Decoding is Memory-Bound: Generating tokens one-by-one relies on General Matrix-Vector multiplication (GEMV). This stage is constrained by memory bandwidth, as the processor must constantly fetch weights and KV cache data for each new token.

For single-batch inference—typical in edge or local AI deployments—GEMV operations can account for over 80% of execution time, rising to over 95% for longer sequences.

Architectural Convergence and Attention Efficiency

Modern LLMs have converged on specific components to mitigate these bottlenecks. This includes the use of RMSNorm for stable normalization, RoPE (Rotary Positional Embedding) for positional encoding, and Gated Linear Units (GLU) for feed-forward layers.

Perhaps most significantly, the evolution of attention mechanisms has moved from Multi-Head Attention (MHA) toward more memory-efficient variants like Grouped Query Attention (GQA) and Multi-Query Attention (MQA). These variants reduce the size of the KV cache, which is essential for maintaining performance during the memory-bound generation stage, especially as batch sizes or sequence lengths increase.

Comparison of Multi-Head, Grouped-Query, and Multi-Query Attention mechanisms, showing the reduction in key-value heads.

Conclusion

While LLM architectures have largely converged, their performance is highly sensitive to the deployment environment. Current high-end GPUs are optimized for massive parallel GEMM operations, making them ideal for high-throughput server environments. However, for edge AI where single-batch GEMV operations dominate, there is a clear need for hardware and architectural optimizations specifically targeted at memory bandwidth constraints. Understanding these operational "footsteps" is crucial for the next generation of efficient AI deployment.

Summary updated

Abstract

The advent of the Attention mechanism and Transformer architecture enables contextually natural text generation and compresses the burden of processing entire source information into singular vectors. Based on these two main ideas, model sizes gradually increases to accommodate more precise and comprehensive information, leading to the current state-of-the-art LLMs being very large, with parameters around 70 billion. As the model sizes are growing, the demand for substantial storage and computational capacity increases. This leads to the development of high-bandwidth memory and accelerators, as well as a variety of model architectures designed to meet these requirements. We note that LLM architectures have increasingly converged. This paper analyzes how these converged architectures perform in terms of layer configurations, operational mechanisms, and model sizes, considering various hyperparameter settings. In this paper, we conduct a concise survey of the history of LLMs by tracing the evolution of their operational improvements. Furthermore, we summarize the performance trends of LLMs under various hyperparameter settings using the RTX 6000, which features the state-of-the-art Ada Lovelace architecture. We conclude that even the same model can exhibit different behaviors depending on the hyperparameters or whether it is deployed in server or edge environments.
BibTeX
@misc{kim2024survey,
  author = {Seongho Kim and Jihyun Moon and Juntaek Oh and Insu Choi and Joon-Sung Yang},
  title = {{Survey and Evaluation of Converging Architecture in LLMs based on Footsteps of Operations}},
  year = {2024},
  eprint = {2410.11381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.11381},
}
Figure