Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021)

Royi Lachmy, Ziyu Yao, Greg Durrett, Milos Gligoric, Jessy Junyi, R. Li, Junayed Mahmud, Fahim Faisal, Raihan Islam Arnob, Antonios Anastasopoulos, Kevin Moran, Aakash Bansal, Sakib Haque, Collin McMillan, Kent Beck, Mike Beedle, Arie van Bennekum, Alis- Tair Cockburn, Ward Cunningham, Martin G. Fowler, James Grenning, Jim Highsmith, A. Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert Martin, Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, M. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov, Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, A. Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Li- Dong Zhou, Linjun Shou, Long Zhou, Michele Tu- Fano, Ming Gong, Ming Zhou, Nan Duan, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Johannes Villmow, Jonas Depoix, Adrian Ulges, Artem Popov, Dmitrii Orekhov, Denis Litvinov, Nikolay Korolev, Gleb Morgachev, Gabriel Orlanski, Alex Gittens, Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama, Hadi M Moein Almasi, Gordon Hemmati, Andrea Fraser, Janis Arcuri, Benefelds, Uri Alon, Meital Zilberstein, Eran Yahav, Siyuan Jiang, Ameer Armaly, Shangqing Liu, Cuiyun Gao, Sen Chen, Nie Lun Yiu, Yang Liu, Zhongxin Liu, Xin Xia, Ahmed Hassan, David Lo, Zhenchang Xing, Xinyu WangOpenalex Unknown 2021 · DOI · OpenAlex

Abstract

The 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), held in conjunction with ACL-IJCNLP 2021, marks a pivotal moment in the intersection of artificial intelligence and software development. This proceedings summary explores the shift toward large-scale transformer models designed to automate documentation, testing, and code generation, while highlighting the critical need for realistic datasets and robust evaluation metrics that reflect the complexities of professional programming.

Introduction: Bridging the Semantic Gap

Software engineering is fundamentally a translation task: converting abstract human intent (e.g., "sort this list by date") into rigid, executable logic. This "semantic gap" has traditionally required manual, error-prone effort for documentation, testing, and code search. The inaugural NLP4Prog workshop addresses the challenges of building Natural Language Interfaces for Software Engineering, specifically focusing on the "Reality Gap"—the discrepancy between clean, curated academic datasets and the messy, context-dependent code found in the wild.

Methodological Innovations

The research presented at the workshop favors Transformer-based architectures, adapted specifically for the unique structural requirements of source code:

  • Multi-task Learning: Models like CoTexT utilize a T5-based encoder-decoder architecture trained simultaneously on code summarization, refinement, and generation. This approach builds a more robust understanding of programming logic compared to single-task models.
  • Contextual Benchmarking: The CONTEST framework introduces a method to feed "context code" (such as recursively called methods) into a transformer, allowing the model to predict unit test assertions with significantly higher accuracy.
  • Advanced Renaming Algorithms: For the task of variable renaming in decompiled code, the DIRECT model introduces an "Identifier Token Coefficient" into its loss function, forcing the model to prioritize naming accuracy over generic syntax.
  • Hybrid Baselines: Interestingly, research also demonstrated that optimized traditional methods, such as Bag-of-Words (BM25), can still outperform complex neural networks in specific semantic code search tasks, suggesting that model complexity is not always a proxy for performance.

New Benchmarks for Real-World Programming

A core contribution of the workshop is the release of several significant datasets designed to move beyond "toy" examples:

  • SEDE: A collection of 12,023 real-world Stack Overflow questions paired with SQL queries. This dataset is designed to be significantly more difficult than standard academic benchmarks like Spider.
  • CONTEST: A large-scale dataset (365k datapoints) for unit test completion that features full project context.
  • Shellcode_IA32: A security-focused dataset containing 3,200 assembly instructions with natural language descriptions for automatic shellcode generation.
  • Funcom: A curated dataset of 2.1 million Java methods used to evaluate the effectiveness of code summarization.

Key Insights and Results

The workshop's findings emphasize that context is king. In unit testing tasks, adding surrounding project context improved model performance by 10.1 BLEU points, proving that models cannot understand a function in isolation.

Performance results were strong across several categories:

  • CodeBERT emerged as a dominant baseline, achieving a BLEU-4 score of 24.15 on Java summarization, significantly outperforming older RNN-based models.
  • The DIRECT model achieved a 20% improvement over previous state-of-the-art models in variable renaming by leveraging bidirectional transformer context.
  • CoTexT set new state-of-the-art results on several CodeXGLUE tasks by pre-training on both bimodal (text + code) and unimodal (code only) data.

However, a "Reality Check" remains: models that score 80-90% on academic SQL benchmarks often drop to ~50% when tested on the "wild" data in the SEDE dataset. This suggests that models may be "cheating" by extracting code directly from natural language prompts rather than demonstrating deep logical understanding.

Limitations and Future Directions

Despite these advancements, significant hurdles remain. Current evaluation metrics like BLEU and ROUGE measure text similarity rather than whether code is executable or logically sound. Future work calls for "execution-based" evaluation and the integration of static analysis (such as Abstract Syntax Trees) with the "soft" logic of Transformers to ensure generated code follows strict language rules. Furthermore, models still struggle to generalize to entirely new software projects not seen during the training phase.

Summary updated

Abstract

Welcome to NLP4Prog, the First Workshop on Natural Language Processing for Programming, colocated with ACL-IJCNLP 2021 online.
BibTeX
@inproceedings{lachmy2021proceedings,
  author = {Royi Lachmy and Ziyu Yao and Greg Durrett and Milos Gligoric and Jessy Junyi and R. Li and Junayed Mahmud and Fahim Faisal and Raihan Islam Arnob and Antonios Anastasopoulos and Kevin Moran and Aakash Bansal and Sakib Haque and Collin McMillan and Kent Beck and Mike Beedle and Arie van Bennekum and Alis- Tair Cockburn and Ward Cunningham and Martin G. Fowler and James Grenning and Jim Highsmith and A. Hunt and Ron Jeffries and Jon Kern and Brian Marick and Robert Martin and Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and M. Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke Zettlemoyer and Veselin Stoyanov and Shuai Lu and Daya Guo and Shuo Ren and Junjie Huang and A. Svyatkovskiy and Ambrosio Blanco and Colin B. Clement and Dawn Drain and Daxin Jiang and Duyu Tang and Li- Dong Zhou and Linjun Shou and Long Zhou and Michele Tu- Fano and Ming Gong and Ming Zhou and Nan Duan and Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Johannes Villmow and Jonas Depoix and Adrian Ulges and Artem Popov and Dmitrii Orekhov and Denis Litvinov and Nikolay Korolev and Gleb Morgachev and Gabriel Orlanski and Alex Gittens and Gabriel Orlanski and Alex Gittens and Takuya Akiba and Shotaro Sano and Toshihiko Yanase and Takeru Ohta and Masanori Koyama and Hadi M Moein Almasi and Gordon Hemmati and Andrea Fraser and Janis Arcuri and Benefelds and Uri Alon and Meital Zilberstein and Omer Levy and Eran Yahav and Siyuan Jiang and Ameer Armaly and Collin Mcmillan and Siyuan Jiang and Collin Mcmillan and Shangqing Liu and Cuiyun Gao and Sen Chen and Nie Lun Yiu and Yang Liu and Zhongxin Liu and Xin Xia and Ahmed Hassan and David Lo and Zhenchang Xing and Xinyu Wang},
  title = {{Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021)}},
  booktitle = {Openalex Unknown},
  year = {2021},
  doi = {10.18653/v1/2021.nlp4prog-1},
}
Figure