Practitioners' Expectations on Code Completion

Chaozheng Wang, Junhao Hu, Cuiyun Gao, Yu Jin, Tao Xie, Hailiang Huang, Zhenyu Lei, Yuetang DengarXiv 2023 · DOI · OpenReview

Abstract

While AI-powered code completion is booming in research, there is a significant disconnect between what researchers build and what software engineers actually need. This study bridges that gap through a large-scale analysis of 599 practitioners, revealing that developers prioritize "skeleton prediction" and grammatical correctness over the "next-line" predictions and BLEU scores favored by current academic literature.

Introduction

Code completion has evolved from simple static analysis to sophisticated Deep Learning (DL) and Pre-trained Language Models (PLMs) like GitHub Copilot. However, most research is driven by model performance on standard benchmarks rather than user experience. The authors identify a "black box" in the field: we don't actually know if practitioners are satisfied with current tools or if researchers are solving the right problems. By investigating these expectations, this paper aims to provide a roadmap for the next generation of developer tools.

Methodology

The authors employed a mixed-methods empirical approach consisting of three distinct stages:

  1. Semi-structured Interviews: 15 professionals (averaging 8.4 years of experience) were interviewed to identify core practices, pain points, and expectations.
  2. Exploratory Survey: 599 practitioners across 18 IT companies (including Microsoft, Intel, and Tencent) were surveyed to quantitatively validate the interview findings.
  3. Literature Review: 26 high-impact papers on code completion published between 2012 and 2022 in premier venues (ICSE, ASE, NeurIPS) were analyzed to compare "state-of-the-art" research against practitioner demands.

Practitioner Expectations and the Research Gap

A primary finding of this study is the mismatch between research focus and developer needs. While researchers focus heavily on "next-line prediction," practitioners are more eager for skeleton prediction (generating the structure of classes/methods) and API argument recommendation.

Practitioner preferences for different types of code completion tasks, categorized by token-level and statement-level.

As shown in the figure above, there is a high demand for "Path completion" (C3), yet zero of the reviewed research papers addressed this task. Furthermore, the study found that for token-level completion (the "as-you-type" experience), 85% of users expect results within 200 milliseconds. For statement-level completion, the tolerance increases to 2 seconds.

Tool Satisfaction and Pain Points

Despite the perceived importance of code completion (88% agreement), only 36% of practitioners are satisfied with their current tools. IDE built-in tools remain the most popular (96% usage), while third-party AI plugins like Copilot and IntelliCode had lower adoption rates at the time of the study.

Practitioner satisfaction levels and specific pain points regarding current code completion tools.

The survey data highlights that the biggest "deal-breakers" for developers are erroneous completions (45%) and painful tool installation (44%). This underscores a critical need for robustness; a code snippet that is 90% "similar" to the ground truth but contains a syntax error is often more frustrating than no completion at all.

The Metric Mismatch

Academics frequently use BLEU and Edit Similarity to evaluate code. Practitioners, however, find these metrics nearly useless. They prioritize grammatical correctness and readability. There is also a split in the community regarding completion length: 46% prefer high-accuracy short completions, while 40% prefer longer completions even if they require manual editing. This suggests that future tools should offer configurable "greediness" settings to accommodate different developer workflows.

Conclusion and Future Directions

The study calls for a shift in research priorities. Instead of chasing marginal gains in BLEU scores, researchers should focus on:

  • Robustness: Ensuring small input changes do not break the model.
  • Offline-first Deployment: Prioritizing privacy and stability for professional environments.
  • Personalized Completion: Developing models that learn and adapt to a specific developer's habits and project context.

By aligning research objectives with these practitioner-centric requirements, the community can move toward tools that provide genuine utility rather than just impressive benchmark performance.

Summary updated

Abstract

Code completion has become a common practice for programmers during their daily programming activities. It aims at automatically predicting the next tokens or lines that the programmers tend to use. A good code completion tool can substantially save keystrokes and improve the programming efficiency for programmers. Recently, various techniques for code completion have been proposed for usage in practice. However, it is still unclear what are practitioners' expectations on code completion and whether existing research has met their demands. To fill the gap, we perform an empirical study by first interviewing 15 practitioners and then surveying 599 practitioners from 18 IT companies about their expectations on code completion. We then compare the practitioners' demands with current research via conducting a literature review of papers on code completion published in premier publication venues from 2012 to 2022. Based on the comparison, we highlight the directions desirable for researchers to invest efforts towards developing code completion techniques for meeting practitioners' expectations.
BibTeX
@misc{wang2023practitioners,
  author = {Chaozheng Wang and Junhao Hu and Cuiyun Gao and Yu Jin and Tao Xie and Hailiang Huang and Zhenyu Lei and Yuetang Deng},
  title = {{Practitioners' Expectations on Code Completion}},
  year = {2023},
  eprint = {2301.03846},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.03846},
}
Figure