The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures

Shyam Agarwal, Olga Lvova, Agnia Sergeyuk, Anastasiia Serova, Timofey Bryksin, Thomas Zimmermann, Bogdan VasilescuarXiv 2026 · arXiv

TL;DR

Is "token spend" the new "Lines of Code"? This case study of 404 JetBrains engineers reveals that AI token usage is a poor proxy for productivity: doubling an engineer's spend correlates with only a 10% increase in commits. Instead of measuring output, token spend primarily reflects an individual's personal work style and their use of AI for non-coding tasks like research and planning.

What Problem Does This Paper Address?

As organizations rush to adopt AI coding agents, managers are desperate for metrics to justify the cost and measure the "return on investment" (ROI). Some companies have resorted to "tokenmaxxing"—creating leaderboards for top token users or setting minimum spending targets—under the assumption that more tokens equals more engineering value.

However, the relationship between the resources an AI consumes (tokens) and the value an engineer produces (shipped features, bug fixes) has remained largely anecdotal. This paper investigates whether token spend is a valid productivity metric or if it’s simply the "Lines of Code" of the AI era: easy to measure, easy to game, and disconnected from actual impact.

Methods

The researchers conducted a longitudinal study at JetBrains, linking two primary data sources over a five-week period:

  1. Gateway Telemetry: They tracked 3.1 million agent calls from 404 engineers, recording timestamps, models used, and token counts.
  2. Repository Data: They matched this telemetry to the engineers' actual output—specifically, commit counts and sizes.

To analyze the data, they used Poisson regression with a "within-person" approach (comparing an engineer’s busy weeks to their own slow weeks) and a "between-person" approach (comparing heavy users to light users). They also analyzed an anonymous internal survey of 412 employees to understand what engineers were actually doing with those tokens.

Datasets

  • JetBrains Internal Telemetry: 5 weeks of metered token usage for 404 engineers.
  • JetBrains Repository Logs: Commit history for the same 584 engineers (including those who didn't use AI).
  • Company AI Survey: Responses from 412 employees (72% in technical roles) regarding their AI usage habits and perceived value.

Key Insights and Ideas

  • The "Stable Trait" Phenomenon: Token spend is largely a property of the engineer, not the task. "Who" the engineer is accounted for 79% of the variation in weekly tokens. Some engineers are simply "heavy users" regardless of whether they are having a high-output or low-output week.
  • The Massive Variance Gap: Within groups of engineers producing the exact same number of commits, the 90th percentile of token spend was 39 to 240 times higher than the 10th percentile. This suggests that two engineers can be equally productive while one consumes hundreds of times more AI resources.
Figure 1: Massive variance in spend for similar commit counts.
  • AI as a Swiss Army Knife: Engineers aren't just using agents to write code. 89% of survey respondents reported using agents for non-coding tasks: data analysis (52%), brainstorming (49%), research (44%), and planning (39%). Token spend measures this "invisible" work that never shows up in a git log.
  • Weak Coupling: The link between spend and output is incredibly loose. Doubling an engineer's spend is associated with only about 10% more commits.
Figure 3: The weak slope between tokens and commits.

Key Results

  • Predictive Power: Traditional work metrics (commits, lines changed) explained only 38% of the variation in spend between different engineers and a meager 13% of the week-to-week variation for a single engineer.
  • Non-Coding Value: Engineers who used AI for a broader range of tasks (2+ non-coding activities) reported saving significantly more time (median 10 hours/week) than those who used it only for coding (median 3 hours/week).
  • The Cost of "AI Slop": 55% of respondents reported spending time (mean 1.6 hours/week) reviewing or fixing "AI-generated material" from colleagues, highlighting a hidden organizational cost.
Figure 4: Factors explaining variation in token use.

Limitations and Future Work

  • Single-Company Study: The data comes from JetBrains, a company that builds developer tools. Their engineers might use AI differently than those in other industries (e.g., finance or healthcare).
  • Commits as a Proxy: The authors acknowledge that commit counts are a noisy measure of productivity that misses code quality, architectural design, and mentorship.
  • Short Duration: The five-week window is too short to observe long-term trends or the "learning curve" of AI adoption.

Future Work: The authors suggest that instead of measuring tokens, organizations should record the intent of AI sessions (e.g., "researching a bug" vs. "writing a test") at the gateway level to better link spend to specific outcomes.

Summary updated

Abstract

Organizations use engineers' token spend to monitor AI adoption, set budgets, and measure productivity, but the metric's relationship with engineering output is unclear. We link five weeks of token usage in coding agents to the commits of 404 engineers at JetBrains, and analyze a company survey of 412 employees. We find that token spend varied mostly between engineers, and which engineers spent the most stayed largely the same over the five weeks. Among weeks with similar commit counts, the 90th percentile of spend was 39 to 240 times the 10th. Within an engineer, doubling spend was associated with about 10% more commits. Available work metrics explained 38% of the variation in spend between engineers and 13% of the week-to-week variation within an engineer. In the survey, respondents in technical roles reported in nearly nine of ten agent evaluations using agents for tasks beyond coding, such as analysis, research, and planning. Token spend therefore measures how much engineers use agents across many kinds of work and not the outcome and value of that work. We recommend using token spend at most for resource accounting, and evaluating agent-assisted work with evidence about tasks and outcomes.
BibTeX
@misc{agarwal2026ai,
  author = {Shyam Agarwal and Olga Lvova and Agnia Sergeyuk and Anastasiia Serova and Timofey Bryksin and Thomas Zimmermann and Bogdan Vasilescu},
  title = {{The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures}},
  year = {2026},
  eprint = {2611.13396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13396},
}
Figure