Natural Language to Code Generation for Interactive Data Science

Pengcheng YinOpenReview 2023 · arXiv · OpenReview

TL;DR

This paper introduces ARCADE, a new benchmark for evaluating natural language to code generation in interactive data science notebooks, specifically for the pandas library. They also present PaChiNCo, a large language model fine-tuned for this task, which significantly outperforms existing models. The work highlights the challenges of context-aware code generation in notebooks and explores prompting strategies to improve model performance and output quality.

What Problem Does This Paper Address?

Data scientists heavily rely on interactive computational notebooks (like Jupyter) for data wrangling and analysis. While AI pair programmers can assist by generating code from natural language (NL) instructions, evaluating their performance in this complex, multi-turn, and context-rich environment is challenging. Existing benchmarks often lack the realistic multi-modal context (code, execution states, previous interactions) and complex dependencies found in real-world notebook usage. This paper aims to bridge this gap by creating a more realistic benchmark and a specialized model to tackle this problem.

Methods

The authors introduce ARCADE, a benchmark comprising 1,082 NL-to-code generation problems specifically for the pandas data analysis framework within Python notebooks. ARCADE is designed to capture the nuances of interactive notebooks, including:

  • Multi-turn interactions: Problems are often sequential, with later intents building on previous ones.
  • Rich multi-modal context: Models must consider existing notebook cells, their execution states, and previous turns of interaction.
  • Complex dependencies: Problems can share execution contexts, form coherent turns, or exhibit non-trivial long-range dependencies.

To address this challenge, they develop PaChiNCo, a 62 billion parameter code language model. PaChiNCo is built upon Google's PaLM model and undergoes a two-stage fine-tuning process: first on a large corpus of Python source code, and then on a collection of Jupyter notebooks. They also explore few-shot prompting strategies, including step-by-step decomposition and inline NL explanations, to guide the model towards generating more diverse, understandable, and explainable code.

Datasets

  • ARCADE Benchmark: A newly created benchmark consisting of 1,082 NL-to-code generation problems across 136 notebooks, covering data wrangling and exploratory data analysis tasks using pandas. It includes both problems derived from existing GitHub notebooks and new problems created from scratch using recent ML datasets.
  • Training Data for PaChiNCo: A large dataset of Python source code files and Jupyter notebooks from GitHub, filtered for permissive licenses. This dataset is approximately 63.7 billion tokens of Python code and 9.6 billion tokens from Jupyter notebooks.

Key Insights and Ideas

  • Notebooks are complex environments: The sequential nature, rich context, and interdependencies within notebooks make NL-to-code generation significantly harder than single-turn tasks.
  • Context is king: Models need to understand not just the current NL intent but also the history of the notebook, including variable states and previous code executions, to generate correct code.
  • Under-specified intents are common: Real-world data science intents are often ambiguous, requiring models to handle uncertainty and potentially generate multiple valid solutions.
  • Step-by-step decomposition and explanation improve models: Prompting models to break down problems and explain their reasoning leads to better accuracy, more diverse solutions, and more understandable code, especially for complex tasks.
  • Domain-specific fine-tuning is crucial: A general-purpose code model fine-tuned on Python code and then specifically on Jupyter notebooks (PaChiNCo) significantly outperforms public code LMs on this specialized task.

Key Results

  • PaChiNCo outperforms public LMs: The proposed PaChiNCo model achieves state-of-the-art performance on the ARCADE benchmark, significantly outperforming existing large code language models. For instance, on the "New Tasks" split of ARCADE, PaChiNCo achieves a pass@30 of 39.8%, compared to CODEGENmono 16B at 26.3% and INCODER 6B at 15.8%.
  • Few-shot prompting enhances performance: Step-by-step prompting strategies, especially when combined with NL explanations, improve the accuracy (pass@k) and diversity of generated code. For example, step-by-step prompting with explanations led to a significant improvement in pass@30 on the "New Tasks" split compared to the baseline.
  • Context length matters: Increasing the amount of notebook context provided to the model leads to improved performance, with diminishing returns after a certain point.
  • Schema descriptions are vital: Including NL descriptions of DataFrame schemas in prompts is crucial for understanding NL intents and generating correct code.

Limitations and Future Work

  • Data leakage concerns: While efforts were made to mitigate data leakage, especially for the "New Tasks" split, the possibility of some overlap with training data for existing models remains a concern.
  • Schema understanding for intermediate DataFrames: Current models struggle with inferring the schema of intermediate DataFrames, leading to errors like KeyErrors and AttributeErrors. Future work could focus on developing LLMs with longer context windows or better mechanisms for handling such information.
  • Improving instruction following: While PaChiNCo performs well, there's still room for improvement in its ability to precisely follow NL intents, especially when dealing with under-specified instructions.
  • Broader task spectrum: The current benchmark focuses on data wrangling and EDA. Future work could expand ARCADE to include other data science tasks like model fitting.
  • More sophisticated evaluation: While fuzzy output matching is used, exploring more robust evaluation metrics that better capture functional equivalence for complex data structures is an ongoing area.
  • Interactive code generation: The paper focuses on generating code for a given cell. Future work could explore more interactive scenarios where the model collaborates with the user in real-time.

Summary updated

Abstract

Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions.
BibTeX
@misc{yin2023natural,
  author = {Pengcheng Yin},
  title = {{Natural Language to Code Generation for Interactive Data Science}},
  year = {2023},
  eprint = {2212.09248},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.09248},
  howpublished = {OpenReview},
}
Figure