Abstract
While AI-powered code completion is booming in research, there is a significant disconnect between what researchers build and what software engineers actually need. This study bridges that gap through a large-scale analysis of 599 practitioners, revealing that developers prioritize "skeleton prediction" and grammatical correctness over the "next-line" predictions and BLEU scores favored by current academic literature.
Introduction
Code completion has evolved from simple static analysis to sophisticated Deep Learning (DL) and Pre-trained Language Models (PLMs) like GitHub Copilot. However, most research is driven by model performance on standard benchmarks rather than user experience. The authors identify a "black box" in the field: we don't actually know if practitioners are satisfied with current tools or if researchers are solving the right problems. By investigating these expectations, this paper aims to provide a roadmap for the next generation of developer tools.
Methodology
The authors employed a mixed-methods empirical approach consisting of three distinct stages:
- Semi-structured Interviews: 15 professionals (averaging 8.4 years of experience) were interviewed to identify core practices, pain points, and expectations.
- Exploratory Survey: 599 practitioners across 18 IT companies (including Microsoft, Intel, and Tencent) were surveyed to quantitatively validate the interview findings.
- Literature Review: 26 high-impact papers on code completion published between 2012 and 2022 in premier venues (ICSE, ASE, NeurIPS) were analyzed to compare "state-of-the-art" research against practitioner demands.
Practitioner Expectations and the Research Gap
A primary finding of this study is the mismatch between research focus and developer needs. While researchers focus heavily on "next-line prediction," practitioners are more eager for skeleton prediction (generating the structure of classes/methods) and API argument recommendation.
As shown in the figure above, there is a high demand for "Path completion" (C3), yet zero of the reviewed research papers addressed this task. Furthermore, the study found that for token-level completion (the "as-you-type" experience), 85% of users expect results within 200 milliseconds. For statement-level completion, the tolerance increases to 2 seconds.
Tool Satisfaction and Pain Points
Despite the perceived importance of code completion (88% agreement), only 36% of practitioners are satisfied with their current tools. IDE built-in tools remain the most popular (96% usage), while third-party AI plugins like Copilot and IntelliCode had lower adoption rates at the time of the study.
The survey data highlights that the biggest "deal-breakers" for developers are erroneous completions (45%) and painful tool installation (44%). This underscores a critical need for robustness; a code snippet that is 90% "similar" to the ground truth but contains a syntax error is often more frustrating than no completion at all.
The Metric Mismatch
Academics frequently use BLEU and Edit Similarity to evaluate code. Practitioners, however, find these metrics nearly useless. They prioritize grammatical correctness and readability. There is also a split in the community regarding completion length: 46% prefer high-accuracy short completions, while 40% prefer longer completions even if they require manual editing. This suggests that future tools should offer configurable "greediness" settings to accommodate different developer workflows.
Conclusion and Future Directions
The study calls for a shift in research priorities. Instead of chasing marginal gains in BLEU scores, researchers should focus on:
- Robustness: Ensuring small input changes do not break the model.
- Offline-first Deployment: Prioritizing privacy and stability for professional environments.
- Personalized Completion: Developing models that learn and adapt to a specific developer's habits and project context.
By aligning research objectives with these practitioner-centric requirements, the community can move toward tools that provide genuine utility rather than just impressive benchmark performance.
Summary updated