TL;DR
This paper provides a comprehensive taxonomy and historical timeline of Large Language Model (LLM) evolution, tracing the path from the 2017 Transformer breakthrough to projected 2025 releases like GPT-5 and Claude 4. It highlights a critical shift in AI design: the move away from raw parameter scaling toward a sustainable balance between architectural complexity and high-quality data to achieve stable reasoning and better alignment.
What Problem Does This Paper Address?
The rapid proliferation of LLMs has created a "fog of war" for researchers and engineers trying to track the field's trajectory. While models are growing more capable, there is a lack of thorough, unified taxonomies that categorize models by their architecture (encoder-only vs. decoder-only), modality (text vs. multimodal), and hardware requirements.
Furthermore, the industry is hitting a wall with "traditional scaling laws." The authors address the challenge of sub-scaling behavior, where simply increasing model size and data volume yields diminishing returns. They explore how the field is grappling with the massive computational, energy, and storage costs required to deploy these models in real-world scenarios.
Methods
The authors conducted a systematic literature review and metadata extraction of approximately 4,000 records spanning 2017 to 2024. Their methodology involved:
- Taxonomy Construction: Categorizing models across three axes: Modality (text, audio, vision), Architecture (encoder-only, decoder-only, encoder-decoder), and Parameter Scale (ranging from "Small" <100M to "HXL" >100B).
- Metadata Curation: Extracting specific quantitative data for each major model, including parameter counts, pre-training data scales, reported compute (FLOPs), weight sizes in bytes, and the specific hardware clusters used (e.g., Nvidia A100s vs. TPU v4s).
- Timeline Analysis: Mapping the transition from early statistical n-grams and RNNs to the current era of Mixture-of-Experts (MoE) and "Thinking" models.
Datasets
The paper does not introduce a new benchmark dataset; instead, it catalogs the massive training corpora used by state-of-the-art models. Key datasets mentioned include:
- Books Corpus & Wikipedia: Used for early models like BERT.
- Common Crawl & Web/Vision/Code Corpora: Used for massive models like Gemini 1.5 and LLaMA 3.
- Proprietary Corpora: The authors note a growing trend toward "Undisclosed" or proprietary datasets for flagship models like GPT-4 and Claude 3.
- Scale: The review tracks the explosion in data scale, from BERT's relatively modest text corpus to LLaMA 3’s 15 trillion tokens and DeepSeek-V3’s 14.8 trillion tokens.
Key Insights and Ideas
- The Efficiency Pivot: The most successful current models aren't just the biggest; they are the ones that optimize the ratio between architectural complexity and data quality. This leads to stronger reasoning stability without a proportional spike in compute load.
- The Storage Bottleneck: We often talk about compute, but storage is a massive hurdle. For example, storing the FP16 weight matrices for a 175B parameter model like GPT-3 requires ~350 GB of space, making deployment on edge or low-end hardware nearly impossible without aggressive compression.
- Sub-scaling Behavior: The authors highlight recent evidence that the gains from traditional scaling laws are beginning to diminish as datasets become increasingly large, suggesting we may need a paradigm shift in how we "teach" these models.
- The Rise of Mixture-of-Experts (MoE): As seen in Figure 2, the industry is moving toward MoE architectures (like GPT-4 and DeepSeek-V3) to maintain high parameter counts while keeping "active" parameters—and thus inference costs—manageable.
Key Results
The paper provides a detailed breakdown of model specifications that serves as a reference for the current state of the art.
Key quantitative highlights include:
- Compute Intensity: GPT-3 required $3.64 \times 10^{23}$ FLOPs, while Gemini 1.5 is estimated to have reached $\sim 10^{24}$ FLOPs.
- Hardware Evolution: The transition from Nvidia V100s for BERT to massive H800 GPU clusters for DeepSeek-R1 and Azure AI Superclusters for GPT-4.
- Multimodal Shift: The latest generation (2024-2025) has moved almost entirely toward multimodal transformers capable of handling text, image, audio, and video simultaneously.
Limitations and Future Work
The authors identify several "open problems" that remain unsolved despite the scaling:
- Hallucination & Reliability: Even with Reinforcement Learning from Human Feedback (RLHF), models still produce contextually wrong but confident-sounding results.
- Quantization Degradation: While quantization (compressing weights to <8 bits) helps with deployment, it often leads to significant performance drops that are difficult to mitigate.
- Black-Box Nature: Flagship models like GPT and Gemini remain "black boxes," where the lack of access to hidden states makes it difficult for researchers to understand how they interpret training data.
- Future Directions: The authors suggest that future research must focus on "agentic" reasoning, real-time adaptation to social media trends, and establishing more robust ethical guidelines for AI-generated content detection.
Summary updated