TL;DR
This paper quantifies the long-term progress in Artificial Intelligence (AI) by developing a new Aggregate State of the Art in ML (ASOTA) index and analyzing various output measures. It finds that AI progress exhibits exponential growth, driven by both computational resources (following a modified Moore's Law) and crucially, human intellectual input, challenging the notion that compute alone is the sole driver.
What Problem Does This Paper Address?
Understanding the drivers of progress in Artificial Intelligence (AI) is crucial given its massive potential impact. While many attribute AI's rapid advancement primarily to the exponential growth of computational power (often linked to Moore's Law), this paper questions whether this is the sole factor. It aims to provide a quantitative framework to understand, predict, and potentially modulate AI development by examining the interplay of various inputs and outputs, and to address the lack of a comprehensive measure for aggregate AI progress and usable computational resources.
Methods
The authors employ an economic production function framework, specifically a Cobb-Douglas model, to analyze AI progress. They replace traditional "capital" with "computational resources" and introduce "labor" (human intellectual input) as a key factor. They construct an "Aggregate State of the Art in ML (ASOTA) Index" by standardizing and combining various ML benchmark performance measures. This index is then used alongside traditional measures like patents and publications to model the relationship between inputs (computational resources and labor) and outputs. They also contrast this with ML scaling laws, which describe empirical relationships between compute and model performance for specific tasks.
Datasets
The paper utilizes several datasets for its analysis:
- Traditional AI Output Measures: Number of published papers and patents related to AI.
- ML Benchmarks: Performance data from various ML tasks, including:
- Language Modelling on Penn Treebank (measured by Perplexity).
- Image Classification on ImageNet (measured by Top 5 Accuracy).
- Chess Engine performance (measured by Elo rating).
- New Benchmark: The Aggregate State of the Art in ML (ASOTA) Index, which is constructed by combining and standardizing numerous ML task-dataset combinations and their performance improvements.
- Economic Data: U.S. Bureau of Economic Analysis (BEA) statistics for industry-level data on capital and labor, including compensation of employees and value added.
- Computational Resource Data: Data on computing hardware costs (USD per GFLOP/s) and investment in private fixed assets for computers and peripheral equipment.
- Labor Data: U.S. Census Bureau's IPUMS data for employment in computer systems analysis and computer science occupations.
Key Insights and Ideas
- AI Progress is Exponential, but Slower than Moore's Law: While AI progress, as measured by various metrics including the new ASOTA index, shows exponential growth, its doubling time is significantly longer (around 2.5-3 years for ML benchmarks) than the roughly two-year doubling time for computational resources (Moore's Law).
- Human Input is Crucial: The paper strongly emphasizes the significant contribution of human intellectual input (labor) to AI progress. The high output elasticity of labor (1-α ≈ 0.8) suggests that increases in skilled researchers translate almost directly into research output.
- Two Frameworks, Convergent Insights: The economic production function model and the ML scaling laws, despite their different scopes (macro vs. micro), yield surprisingly similar scaling exponents (α ≈ 0.2 vs. α' ≈ 0.1), lending credibility to both approaches.
- Moore's Law's Nuance: While Moore's Law remains a dominant factor, its role is more nuanced than a simple direct driver. The paper suggests that the growth in computational resources devoted to AI research (ΦAI) may have accelerated in recent years, potentially explaining some discrepancies in output measures.
- "AI Winters" Not Outliers: The historical periods of slowdown in AI research ("AI winters") do not appear as outliers from the proposed economic model's perspective.
Key Results
- The authors developed the Aggregate State of the Art in ML (ASOTA) Index, providing a continuous measure of AI progress.
- A "minimal economic model" based on a Cobb-Douglas production function with computational resources and labor as inputs, and an output elasticity of α = 0.2 for computational resources, fits output measures (papers, patents, ASOTA, LM, IC, Elo) with reasonable accuracy over five decades.
- The model's predictions align well with historical data, offering a better fit than a simple "doubling every two years" extrapolation.
- The labor share of income in the AI research sector is estimated to be 1 - α ≈ 0.8, highlighting its substantial contribution.
- ML benchmarks like ASOTA show doubling times of 2.5 to 3 years, while traditional measures like patents and papers have seen faster growth since 2012.
Limitations and Future Work
- Estimating Computational Resources Devoted to AI (ΦAI): The paper acknowledges that accurately estimating the fraction of computational resources specifically devoted to AI research is an open question, with reasons to believe this fraction has grown substantially in recent years.
- Data Granularity and Definitions: The statistical series used for capital and labor may be broader than strictly AI-related categories, potentially overestimating these factors.
- Autocorrelated Data: The analysis relies on several decades of data, which, while providing a long-term view, can be statistically considered a small sample due to autocorrelation.
- Other Input Factors: The paper suggests exploring other potential input factors, such as the exponential growth of "research productivity" (At) itself, and the impact of AI on productivity in other industries.
- Understanding AI's Impact on AI Research: A key future direction is to understand how AI progress itself influences the productivity of AI research.
Summary updated