Transformer to GPT-5: Advancements, Limitations, and Future Directions in Large Language Model Design

Harsidak Singh Banwait, Chander PrabhaarXiv 2026 · arXiv

TL;DR

This paper provides a comprehensive taxonomy and historical timeline of Large Language Model (LLM) evolution, tracing the path from the 2017 Transformer breakthrough to projected 2025 releases like GPT-5 and Claude 4. It highlights a critical shift in AI design: the move away from raw parameter scaling toward a sustainable balance between architectural complexity and high-quality data to achieve stable reasoning and better alignment.

What Problem Does This Paper Address?

The rapid proliferation of LLMs has created a "fog of war" for researchers and engineers trying to track the field's trajectory. While models are growing more capable, there is a lack of thorough, unified taxonomies that categorize models by their architecture (encoder-only vs. decoder-only), modality (text vs. multimodal), and hardware requirements.

Furthermore, the industry is hitting a wall with "traditional scaling laws." The authors address the challenge of sub-scaling behavior, where simply increasing model size and data volume yields diminishing returns. They explore how the field is grappling with the massive computational, energy, and storage costs required to deploy these models in real-world scenarios.

Methods

The authors conducted a systematic literature review and metadata extraction of approximately 4,000 records spanning 2017 to 2024. Their methodology involved:

  • Taxonomy Construction: Categorizing models across three axes: Modality (text, audio, vision), Architecture (encoder-only, decoder-only, encoder-decoder), and Parameter Scale (ranging from "Small" <100M to "HXL" >100B).
  • Metadata Curation: Extracting specific quantitative data for each major model, including parameter counts, pre-training data scales, reported compute (FLOPs), weight sizes in bytes, and the specific hardware clusters used (e.g., Nvidia A100s vs. TPU v4s).
  • Timeline Analysis: Mapping the transition from early statistical n-grams and RNNs to the current era of Mixture-of-Experts (MoE) and "Thinking" models.

Datasets

The paper does not introduce a new benchmark dataset; instead, it catalogs the massive training corpora used by state-of-the-art models. Key datasets mentioned include:

  • Books Corpus & Wikipedia: Used for early models like BERT.
  • Common Crawl & Web/Vision/Code Corpora: Used for massive models like Gemini 1.5 and LLaMA 3.
  • Proprietary Corpora: The authors note a growing trend toward "Undisclosed" or proprietary datasets for flagship models like GPT-4 and Claude 3.
  • Scale: The review tracks the explosion in data scale, from BERT's relatively modest text corpus to LLaMA 3’s 15 trillion tokens and DeepSeek-V3’s 14.8 trillion tokens.

Key Insights and Ideas

  • The Efficiency Pivot: The most successful current models aren't just the biggest; they are the ones that optimize the ratio between architectural complexity and data quality. This leads to stronger reasoning stability without a proportional spike in compute load.
  • The Storage Bottleneck: We often talk about compute, but storage is a massive hurdle. For example, storing the FP16 weight matrices for a 175B parameter model like GPT-3 requires ~350 GB of space, making deployment on edge or low-end hardware nearly impossible without aggressive compression.
  • Sub-scaling Behavior: The authors highlight recent evidence that the gains from traditional scaling laws are beginning to diminish as datasets become increasingly large, suggesting we may need a paradigm shift in how we "teach" these models.
  • The Rise of Mixture-of-Experts (MoE): As seen in Figure 2, the industry is moving toward MoE architectures (like GPT-4 and DeepSeek-V3) to maintain high parameter counts while keeping "active" parameters—and thus inference costs—manageable.

Key Results

The paper provides a detailed breakdown of model specifications that serves as a reference for the current state of the art.

Summary of LLM architectures and their hardware specifications, showing the progression from BERT (340M parameters) to PaLM (540B parameters).

Key quantitative highlights include:

  • Compute Intensity: GPT-3 required $3.64 \times 10^{23}$ FLOPs, while Gemini 1.5 is estimated to have reached $\sim 10^{24}$ FLOPs.
  • Hardware Evolution: The transition from Nvidia V100s for BERT to massive H800 GPU clusters for DeepSeek-R1 and Azure AI Superclusters for GPT-4.
  • Multimodal Shift: The latest generation (2024-2025) has moved almost entirely toward multimodal transformers capable of handling text, image, audio, and video simultaneously.
Table II: Summary of the current multimodal LLM architecture

Limitations and Future Work

The authors identify several "open problems" that remain unsolved despite the scaling:

  • Hallucination & Reliability: Even with Reinforcement Learning from Human Feedback (RLHF), models still produce contextually wrong but confident-sounding results.
  • Quantization Degradation: While quantization (compressing weights to <8 bits) helps with deployment, it often leads to significant performance drops that are difficult to mitigate.
  • Black-Box Nature: Flagship models like GPT and Gemini remain "black boxes," where the lack of access to hidden states makes it difficult for researchers to understand how they interpret training data.
  • Future Directions: The authors suggest that future research must focus on "agentic" reasoning, real-time adaptation to social media trends, and establishing more robust ethical guidelines for AI-generated content detection.

Summary updated

Abstract

Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) by their extraordinary capabilities, like contextual understanding, few-shot/zero-shot learning, code generation, ability to reason and problem-solving. Recent studies have shown immense work on LLMs. Yet, there is limited work on the thorough taxonomy of LLMs. This study would showcase a detailed classification of LLMs, starting with the earlier models like BERT and leading to LLMs like GPT, Gemini, Claude and Grok. We would delve into topics like datasets used to train LLMs, computational power used during training, their hardware specifications, architecture and parameters. Further, we would also look at the limitations, computational challenges and open problems as well. This study would primarily focus on the timeline of LLMs, signifying their growth over the period. Findings reveal a clear pattern: models that optimize the ratio between architectural complexity and data quality exhibit stronger reasoning stability and better alignment, without proportionally increasing the computational load. This desired balance between scaling and sustainability is the defining characteristic of the current generation of AI.
BibTeX
@misc{banwait2026transformer,
  author = {Harsidak Singh Banwait and Chander Prabha},
  title = {{Transformer to GPT-5: Advancements, Limitations, and Future Directions in Large Language Model Design}},
  year = {2026},
  eprint = {2610.34281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34281},
}
Figure