Natural Language to Code Generation in Interactive Data Science Notebooks

Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, Charles Sutton

Introduction

Data science is inherently iterative, characterized by a conversational workflow between researchers and data within computational notebooks like Jupyter. While Large Language Models (LLMs) excel at generating standalone code snippets, they often struggle with the stateful, multi-turn nature of real-world data science projects. This paper introduces ARCADE, a benchmark designed for these realistic settings, and PACHINCO, a 62B parameter model optimized for notebook-based code generation.

Challenges in Notebook-Based Code Generation

Traditional benchmarks like HumanEval focus on isolated functions. However, data science notebooks present unique challenges:

  1. Multi-turn Interaction: Code cells often reference variables or states established in previous turns.
  2. Implicit Context: The intent of a command frequently depends on the underlying data schema, which may not be explicitly stated in the current cell.
  3. Under-specification: User comments are often brief and "ephemeral" (e.g., "plot the distribution"), requiring the model to infer intent from context.

The ARCADE Benchmark

To address these challenges, the authors developed ARCADE, consisting of 1,082 multi-turn Python/pandas problems across 136 notebooks. To mitigate data leakage—a common issue where evaluation data is present in training sets—65% of the benchmark (the "New Tasks" split) was created using recent Kaggle datasets.

Table 1: Comparison of ARCADE with existing datasets, highlighting its focus on multi-turn problems and higher API complexity.

As shown in the comparison above, ARCADE features a higher number of problems per notebook and a greater average number of API calls per problem compared to previous benchmarks like JuICe or DS-1000.

Figure 1: Distribution of pandas API calls in ARCADE, showing the complexity of both existing and new tasks.

The complexity of these tasks is further illustrated by the distribution of pandas API calls, with many problems requiring 4 to 8 distinct calls to reach a solution.

The PACHINCO Model

PACHINCO is a 62B parameter model based on the PaLM architecture. It utilizes a two-stage fine-tuning process:

  1. General Code Pre-training: Training on 64B tokens of Python source code.
  2. Notebook-Specific Fine-tuning: Training on 9.6B tokens from 3.8M Jupyter notebooks to capture the "literate programming" style.

The researchers also introduced Fuzzy Output Matching, a metric that validates solutions based on the resulting dataframe's content rather than strict code equality, and Step-by-Step (SbS) Prompting, which encourages models to decompose tasks into intermediate steps.

Results and Discussion

PACHINCO significantly outperformed existing models. On the "New Tasks" split, PACHINCO achieved a pass@30 of 48.6%, nearly doubling the performance of CodeGen-mono (26.3%). Key findings include:

  • Context Window: Including 3 to 5 previous cells provides the most significant performance boost; beyond 7 cells, returns diminish.
  • Memorization vs. Generalization: The performance drop on "New Tasks" suggests that many LLMs rely on memorization of GitHub data, whereas ARCADE provides a more robust measure of generalization.
  • Schema Awareness: The primary failure mode was "schema blindness," where models failed to track how operations like groupby altered dataframe structures.

Conclusion

The ARCADE benchmark and PACHINCO model represent a significant step toward AI-assisted data science. By focusing on multi-turn context and realistic notebook environments, this work highlights the necessity of schema awareness and task decomposition for future code generation models.

Summary updated

BibTeX
@inproceedings{yin2023natural,
  author = {Pengcheng Yin and Wen-Ding Li and Kefan Xiao and Abhishek Rao and Yeming Wen and Kensen Shi and Joshua Howland and Paige Bailey and Michele Catasta and Henryk Michalewski and Oleksandr Polozov and Charles Sutton},
  title = {{Natural Language to Code Generation in Interactive Data Science Notebooks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.9},
}
Figure