TL;DR
This paper challenges the prevailing narrative that compute access is the primary bottleneck for AI research. Through a large-scale survey of US AI researchers, the authors found that talent and data limitations are more significant constraints, with surprisingly few differences between academic and industry researchers. This suggests that current policy proposals focused solely on subsidizing compute might not be the most effective way to foster equitable AI research.
What Problem Does This Paper Address?
There's a growing concern that unequal access to computational resources is hindering AI research progress and concentrating power in the hands of a few large organizations. This has led to proposals like the National AI Research Resource (NAIRR) in the US, aiming to invest billions in compute access. However, the voices of the AI researchers themselves, who directly experience these constraints, have been largely absent from this conversation. This paper aims to fill that gap by directly asking AI researchers about their experiences and perceptions of resource limitations, specifically comparing compute, data, and talent.
Methods
The researchers conducted a large-scale survey of US-based AI researchers. The survey included 30-35 close-ended questions and one open-ended question, tailored to each respondent's experience. Before deployment, the survey instrument underwent cognitive interviews with AI researchers from both academia and industry to refine its clarity and relevance. The sampling frame was constructed by identifying authors of papers in top AI conferences and journals, as well as AI researchers in industry roles identified through LinkedIn. The survey was distributed in June 2022, with follow-ups to ensure a robust response rate.
Datasets
Not discussed in the paper. The paper focuses on survey data collected from AI researchers.
Key Insights and Ideas
- Compute isn't the main bottleneck for most: The dominant narrative that compute is the primary barrier to AI research is not supported by the majority of researchers surveyed. This is a significant finding that challenges current policy directions.
- Talent and data are the real constraints: Researchers consistently ranked "specialized knowledge, talent, or skills" as the most important factor for project success and as their top priority for additional funding. Data limitations were also cited more frequently than compute limitations when researchers had to reject or abandon projects.
- Academia vs. Industry: Not as different as you might think: Contrary to popular belief, the study found surprisingly few differences in compute usage and concerns between academic and industry researchers. While academics reported spending less on compute, their reported GPU-hour usage was similar. Both groups expressed similar levels of concern about future compute access impacting their contributions.
- High-compute users are more concerned: Researchers who already use large amounts of compute were more likely to express concern about future compute access and to consider compute an important factor in their career decisions. This suggests a self-selection bias, where those already invested in compute-intensive research are more vocal about its limitations.
- Grant funding is preferred over specific resources: When asked about government support, researchers generally preferred grant funding over specific resources like compute or curated data, indicating a desire for flexibility in how they address their unique needs.
Key Results
- Talent is paramount: 90% of respondents rated "specialized knowledge, talent, or skills" as very or extremely important for their most significant project's success, compared to 52% for "large amounts of compute."
- Funding priorities: Roughly half (52%) of researchers would prioritize hiring more researchers or engineers if their project budget doubled, while only about a fifth would prioritize purchasing more compute.
- Project disruptions: Researchers reported rejecting and abandoning projects due to a lack of data or researcher availability more often than from insufficient compute resources.
- Compute's role in past vs. future progress: 59% of respondents agreed that compute was a major driver of AI progress in the past decade. However, only 40% strongly agreed that compute would be a driver of AI progress over the next decade, with "better algorithms" showing a stronger expected future impact.
- Compute use similarity: While academics reported spending significantly less on compute for their most compute-intensive projects, there was no meaningful difference in reported GPU-hours used between academic and industry researchers.
- Compute needs outpacing access for academics: A significantly greater proportion of academics reported that their compute needs have outpaced their access compared to industry researchers.
Limitations and Future Work
The authors acknowledge that their findings regarding specific subgroups (e.g., high-compute users, language modelers, AI startup employees) are based on small sample sizes, and future work could focus on these groups to better understand their unique requirements. The study's scope is limited to AI research within the United States, and future research could explore differences in compute access and its implications among institutions with varying research budgets. Additionally, this paper focuses on AI research, not AI practitioners, and the question of how compute access influences downstream applications of AI remains unaddressed. The authors also note that while compute is a constraint for certain types of AI research (like large foundation models), for most researchers, talent remains a greater constraint.
Summary updated