TL;DR
Is "token spend" the new "Lines of Code"? This case study of 404 JetBrains engineers reveals that AI token usage is a poor proxy for productivity: doubling an engineer's spend correlates with only a 10% increase in commits. Instead of measuring output, token spend primarily reflects an individual's personal work style and their use of AI for non-coding tasks like research and planning.
What Problem Does This Paper Address?
As organizations rush to adopt AI coding agents, managers are desperate for metrics to justify the cost and measure the "return on investment" (ROI). Some companies have resorted to "tokenmaxxing"—creating leaderboards for top token users or setting minimum spending targets—under the assumption that more tokens equals more engineering value.
However, the relationship between the resources an AI consumes (tokens) and the value an engineer produces (shipped features, bug fixes) has remained largely anecdotal. This paper investigates whether token spend is a valid productivity metric or if it’s simply the "Lines of Code" of the AI era: easy to measure, easy to game, and disconnected from actual impact.
Methods
The researchers conducted a longitudinal study at JetBrains, linking two primary data sources over a five-week period:
- Gateway Telemetry: They tracked 3.1 million agent calls from 404 engineers, recording timestamps, models used, and token counts.
- Repository Data: They matched this telemetry to the engineers' actual output—specifically, commit counts and sizes.
To analyze the data, they used Poisson regression with a "within-person" approach (comparing an engineer’s busy weeks to their own slow weeks) and a "between-person" approach (comparing heavy users to light users). They also analyzed an anonymous internal survey of 412 employees to understand what engineers were actually doing with those tokens.
Datasets
- JetBrains Internal Telemetry: 5 weeks of metered token usage for 404 engineers.
- JetBrains Repository Logs: Commit history for the same 584 engineers (including those who didn't use AI).
- Company AI Survey: Responses from 412 employees (72% in technical roles) regarding their AI usage habits and perceived value.
Key Insights and Ideas
- The "Stable Trait" Phenomenon: Token spend is largely a property of the engineer, not the task. "Who" the engineer is accounted for 79% of the variation in weekly tokens. Some engineers are simply "heavy users" regardless of whether they are having a high-output or low-output week.
- The Massive Variance Gap: Within groups of engineers producing the exact same number of commits, the 90th percentile of token spend was 39 to 240 times higher than the 10th percentile. This suggests that two engineers can be equally productive while one consumes hundreds of times more AI resources.
- AI as a Swiss Army Knife: Engineers aren't just using agents to write code. 89% of survey respondents reported using agents for non-coding tasks: data analysis (52%), brainstorming (49%), research (44%), and planning (39%). Token spend measures this "invisible" work that never shows up in a git log.
- Weak Coupling: The link between spend and output is incredibly loose. Doubling an engineer's spend is associated with only about 10% more commits.
Key Results
- Predictive Power: Traditional work metrics (commits, lines changed) explained only 38% of the variation in spend between different engineers and a meager 13% of the week-to-week variation for a single engineer.
- Non-Coding Value: Engineers who used AI for a broader range of tasks (2+ non-coding activities) reported saving significantly more time (median 10 hours/week) than those who used it only for coding (median 3 hours/week).
- The Cost of "AI Slop": 55% of respondents reported spending time (mean 1.6 hours/week) reviewing or fixing "AI-generated material" from colleagues, highlighting a hidden organizational cost.
Limitations and Future Work
- Single-Company Study: The data comes from JetBrains, a company that builds developer tools. Their engineers might use AI differently than those in other industries (e.g., finance or healthcare).
- Commits as a Proxy: The authors acknowledge that commit counts are a noisy measure of productivity that misses code quality, architectural design, and mentorship.
- Short Duration: The five-week window is too short to observe long-term trends or the "learning curve" of AI adoption.
Future Work: The authors suggest that instead of measuring tokens, organizations should record the intent of AI sessions (e.g., "researching a bug" vs. "writing a test") at the gateway level to better link spend to specific outcomes.
Summary updated