TL;DR
This paper presents the first controlled experiment measuring the productivity impact of GitHub Copilot, an AI pair programmer. The study found that developers using Copilot completed a coding task 55.8% faster than those without, suggesting significant productivity gains for AI-assisted software development.
What Problem Does This Paper Address?
The rapid advancement of AI, particularly in code generation, has sparked widespread interest in its potential to boost human productivity. However, concrete, empirical evidence of these productivity gains in professional software development contexts has been scarce. While many studies have explored user perceptions and the potential implications of AI tools for security and education, there's a critical need to quantify the actual impact on developer efficiency. This paper aims to fill that gap by rigorously measuring the productivity difference when developers use GitHub Copilot for a specific programming task.
Methods
The researchers conducted a controlled experiment involving 95 professional programmers recruited through Upwork. Participants were randomly assigned to either a "treatment" group with access to GitHub Copilot or a "control" group without it. Both groups were tasked with implementing a simple HTTP server in JavaScript. The experiment was administered using GitHub Classroom, which allowed for precise measurement of task completion time and success. The primary metric was task completion time, defined as the duration from starting the task to successfully passing all 12 automated tests. The study also collected demographic data and conducted an exit survey to gather self-reported productivity gains and willingness to pay for Copilot.
Datasets
Not discussed in the paper. The study used a custom-designed task and data collected from the experiment participants.
Key Insights and Ideas
- Quantifiable Productivity Boost: The most striking finding is the significant, measurable speedup provided by GitHub Copilot. This moves beyond anecdotal evidence to provide concrete data on AI's impact.
- Heterogeneous Effects Matter: The benefits of Copilot are not uniform. Developers with less experience, those who program more hours daily, and younger developers (25-44) saw the most substantial gains. This suggests AI tools could be particularly valuable for onboarding new developers or supporting those with heavy workloads.
- Willingness to Pay Correlates with Benefit: The study found that developers who used Copilot were willing to pay significantly more for it monthly compared to the control group. This indirect evidence supports the idea that the perceived and actual benefits translate into economic value.
- Self-Reported vs. Revealed Productivity: While participants self-reported a productivity increase, their estimates were lower than the actual measured speedup. This highlights the potential for developers to underestimate the impact of AI tools.
Key Results
- The treatment group using GitHub Copilot completed the task 55.8% faster than the control group.
- The 95% confidence interval for this improvement was [21%, 89%].
- The average completion time for the treated group was 71.17 minutes, compared to 160.89 minutes for the control group.
- The treated group's success rate was 7 percentage points higher than the control group, though this was not statistically significant.
- Developers with less programming experience, those who program more hours per day, and those aged 25-44 benefited the most from Copilot.
- The average self-reported productivity gain was around 35% for both groups, underestimating the measured 55.8% improvement.
- The average willingness to pay for Copilot was $27.25/month** for the treated group, versus **$16.91/month for the control group.
Limitations and Future Work
The authors acknowledge several limitations:
- Standardized Task: The study used a specific, standardized programming task. The generalizability of these findings to other tasks, programming languages, or larger, more complex projects in professional settings needs further investigation.
- Code Quality Not Examined: The study focused solely on productivity (speed) and did not assess the impact of Copilot on code quality, which could have performance and security implications.
- Experiment Population: The participants were freelancers recruited via Upwork, and their demographics (e.g., lower income, high education) might not represent all software developers.
Future work suggested includes:
- Investigating the impact of AI on code quality.
- Exploring how productivity benefits vary across different tasks and languages.
- Examining the broader economic implications of AI-driven productivity gains, including effects on the labor market, employment, and GDP.
- Understanding how job tasks might evolve to incorporate AI-powered developer tools.
Summary updated