Compute Trends Across Three Eras of Machine Learning

Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos

Introduction

The rapid advancement of Machine Learning (ML) over the past decade has transformed fields from computer vision to natural language processing. While progress is driven by a combination of algorithmic innovation, data availability, and computational power, compute remains the most readily quantifiable metric. This paper presents a rigorous historical analysis of training compute across 123 milestone ML models from 1952 to 2022, identifying distinct shifts in the scale and pace of AI development.

Methodology

To map the evolution of ML compute, we curated a dataset of 123 "milestone" systems selected based on historical importance, citation count, or significant state-of-the-art (SotA) advances. For each system, we estimated the total training compute in Floating Point Operations (FLOPs) using two primary methods:

  1. Forward Pass Compute: Calculating FLOPs based on model architecture and the number of training tokens.
  2. Hardware-Time: Estimating compute based on the hardware used (e.g., GPU/TPU type), the number of chips, and total training duration, adjusted for utilization rates.

Using log-linear regression and bootstrapping, we identified statistical "breaks" in the historical trend, revealing three distinct eras of compute growth.

The Three Eras of Compute

Our analysis reveals that the history of ML compute is not a single continuous trend but is divided into three distinct regimes based on the rate of growth and absolute scale.

Figure 1: Training compute (FLOPs) of milestone ML systems from 1952 to 2022, categorized into three eras.

1. The Pre-Deep Learning Era (1952 – 2010)

For nearly six decades, ML compute grew in lockstep with Moore’s Law. During this period, compute requirements doubled approximately every 21.3 months. This suggests that progress was primarily constrained by the pace of hardware improvements.

2. The Deep Learning Era (2010 – 2022)

Around 2010, the trend accelerated dramatically. With the advent of deep learning, compute requirements began doubling every 5.7 months. This pace far outstrips hardware improvements, indicating a massive increase in the resources—both financial and infrastructure—dedicated to training larger models.

3. The Large-Scale Era (2015 – 2022)

In late 2015, a new "flagship" tier emerged. Starting with systems like AlphaGo, these models operate 2 to 3 orders of magnitude above the standard deep learning trend. These projects, typically developed by well-funded corporate labs, represent a significant "step up" in investment.

Figure 3: Detailed view of the Deep Learning and Large-Scale eras (2010-2022), highlighting specific milestone models and the divergence of flagship projects.

Quantitative Results

The following table summarizes the growth characteristics of each era:

Era Period Doubling Time Scale Range (FLOPs)
Pre-Deep Learning 1952 – 2010 ~21.3 months 3e+04 to 2e+14
Deep Learning 2010 – 2022 ~5.7 months 7e+14 to 2e+18
Large-Scale 2015 – 2022 ~9.9 months 4e+21 to 8e+23

Interestingly, while the Large-Scale era operates at a much higher absolute magnitude, its doubling time (~9.9 months) is slower than the general Deep Learning trend (~5.7 months). This may suggest that we are approaching the practical limits of current hardware clusters and research budgets.

Discussion and Limitations

While this dataset provides a comprehensive look at ML compute, several limitations apply. First, the focus on "milestone" models introduces a selection bias toward the largest and most successful projects. Second, FLOP estimates are often based on incomplete hardware data provided in publications. Finally, the study primarily examines English-language academic and corporate literature, which may overlook developments in other regions or undisclosed industrial models.

Conclusion

The transition from a hardware-constrained "Pre-Deep Learning" era to the investment-driven "Deep Learning" and "Large-Scale" eras marks a fundamental shift in AI research. As we move forward, the sustainability of these trends will likely depend on overcoming potential "data walls" and the continued scaling of compute infrastructure.

Summary updated

BibTeX
@inproceedings{sevilla2022compute,
  author = {Jaime Sevilla and Lennart Heim and Anson Ho and Tamay Besiroglu and Marius Hobbhahn and Pablo Villalobos},
  title = {{Compute Trends Across Three Eras of Machine Learning}},
  booktitle = {IJCNN},
  year = {2022},
  doi = {10.1109/ijcnn55064.2022.9891914},
}
Figure