Introduction
Data science is inherently iterative, characterized by a conversational workflow between researchers and data within computational notebooks like Jupyter. While Large Language Models (LLMs) excel at generating standalone code snippets, they often struggle with the stateful, multi-turn nature of real-world data science projects. This paper introduces ARCADE, a benchmark designed for these realistic settings, and PACHINCO, a 62B parameter model optimized for notebook-based code generation.
Challenges in Notebook-Based Code Generation
Traditional benchmarks like HumanEval focus on isolated functions. However, data science notebooks present unique challenges:
- Multi-turn Interaction: Code cells often reference variables or states established in previous turns.
- Implicit Context: The intent of a command frequently depends on the underlying data schema, which may not be explicitly stated in the current cell.
- Under-specification: User comments are often brief and "ephemeral" (e.g., "plot the distribution"), requiring the model to infer intent from context.
The ARCADE Benchmark
To address these challenges, the authors developed ARCADE, consisting of 1,082 multi-turn Python/pandas problems across 136 notebooks. To mitigate data leakage—a common issue where evaluation data is present in training sets—65% of the benchmark (the "New Tasks" split) was created using recent Kaggle datasets.
As shown in the comparison above, ARCADE features a higher number of problems per notebook and a greater average number of API calls per problem compared to previous benchmarks like JuICe or DS-1000.
The complexity of these tasks is further illustrated by the distribution of pandas API calls, with many problems requiring 4 to 8 distinct calls to reach a solution.
The PACHINCO Model
PACHINCO is a 62B parameter model based on the PaLM architecture. It utilizes a two-stage fine-tuning process:
- General Code Pre-training: Training on 64B tokens of Python source code.
- Notebook-Specific Fine-tuning: Training on 9.6B tokens from 3.8M Jupyter notebooks to capture the "literate programming" style.
The researchers also introduced Fuzzy Output Matching, a metric that validates solutions based on the resulting dataframe's content rather than strict code equality, and Step-by-Step (SbS) Prompting, which encourages models to decompose tasks into intermediate steps.
Results and Discussion
PACHINCO significantly outperformed existing models. On the "New Tasks" split, PACHINCO achieved a pass@30 of 48.6%, nearly doubling the performance of CodeGen-mono (26.3%). Key findings include:
- Context Window: Including 3 to 5 previous cells provides the most significant performance boost; beyond 7 cells, returns diminish.
- Memorization vs. Generalization: The performance drop on "New Tasks" suggests that many LLMs rely on memorization of GitHub data, whereas ARCADE provides a more robust measure of generalization.
- Schema Awareness: The primary failure mode was "schema blindness," where models failed to track how operations like
groupbyaltered dataframe structures.
Conclusion
The ARCADE benchmark and PACHINCO model represent a significant step toward AI-assisted data science. By focusing on multi-turn context and realistic notebook environments, this work highlights the necessity of schema awareness and task decomposition for future code generation models.
Summary updated