Towards a High Level Linter for Data Science

Greta Dolcetti, Agostino Cortesi, Caterina Urban, Enea Zaffanella

TL;DR

This paper introduces a novel "high-level" linter for data science code, going beyond traditional syntax checks to identify subtle logical errors and "code smells" specific to data science workflows. By employing an abstract interpretation framework with custom data science-centric abstract datatypes, the tool aims to help data scientists write more robust and reliable code, especially in dynamic languages like Python.

What Problem Does This Paper Address?

The explosive growth of data science has led to widespread adoption of powerful but dynamically typed languages like Python. While these languages offer flexibility, they also make it easier to introduce subtle logical errors that can go undetected until runtime, or worse, lead to incorrect interpretations of results without explicit errors. Existing static analysis tools often focus on low-level programming issues or standard language features, leaving a gap in identifying higher-level, domain-specific mistakes common in data science pipelines (e.g., misinterpreting data types, inappropriate visualization choices, or data leakage during preprocessing). This paper aims to bridge that gap by developing a linter that understands the semantics of data science libraries.

Methods

The core of the proposed solution is a static analysis tool built upon the Abstract Interpretation framework. This framework allows for sound approximations of program behavior, which is crucial for handling the complexity of data science code. The key innovations are:

  • Custom Abstract Datatype System: Instead of relying on standard type systems, the linter defines abstract datatypes that capture the specific characteristics of data science objects, such as DataFrame, Series, RatioSeries (for results of division), and NormSeries (for normalized data).
  • Data Science Library Signatures: The analysis doesn't delve into the implementation of data science libraries (like pandas or scikit-learn). Instead, it uses abstract signatures for library functions to infer how these operations affect the abstract datatypes of variables.
  • Descriptive, Not Prescriptive: The linter is designed to be "descriptive" rather than "prescriptive." It flags potential issues ("code smells") and encourages critical thinking, rather than strictly enforcing rules that might be overly restrictive for data science workflows.
  • Incremental Development: The tool is designed for incremental extension, allowing new abstract datatypes and library function rules to be added as the data science ecosystem evolves.

Datasets

Not discussed in the paper. The paper focuses on the design and methodology of the linter and illustrates its capabilities with code examples from typical data science tasks.

Key Insights and Ideas

  • Beyond Type Errors: The most significant insight is that many critical errors in data science code are not traditional type errors but rather logical mistakes stemming from a misunderstanding or misuse of data types and library functions.
  • The Importance of "Code Smells": The paper highlights "code smells" – indicators of deeper problems – as the primary target for a data science linter. Examples include using a line plot for categorical data or computing the arithmetic mean of speedup ratios.
  • Abstract Interpretation as a Foundation: The choice of Abstract Interpretation is well-suited for this problem due to its ability to handle approximations and incremental refinement, which are essential for building a flexible and precise analysis tool.
  • Descriptive Analysis for Data Scientists: A linter that informs rather than dictates is more appropriate for data scientists, who may not always have a traditional software engineering background. This approach fosters better understanding and adoption.

Key Results

The paper presents a prototype implementation and demonstrates its ability to identify several types of code smells through illustrative examples:

  • Misleading Visualizations: Detecting when a plot type is inappropriate for the underlying data (e.g., line plot for string data).
  • Inappropriate Central Tendency Measures: Identifying when the arithmetic mean is used for data where the geometric mean or median would be more suitable (e.g., ratios, ordinal data).
  • Misuse of Scaled Data: Flagging the computation of means on normalized data, which can lead to misleading results.
  • Potential Data Leakage: Highlighting scenarios where data normalization might occur before splitting train/test sets, leading to leakage.
  • Incorrect Return Values: Detecting when functions that are expected to return a value (e.g., fillna with inplace=True) do not, potentially leading to assignment errors.

The paper shows concrete examples of warnings generated by the prototype, such as a "Gmean Warning" suggesting the use of geometric mean for speedup values.

Limitations and Future Work

  • Current Prototype Limitations: The current prototype primarily handles Python code and focuses on a subset of pandas and scikit-learn operations. It does not consider arbitrary code cell execution in Jupyter notebooks.
  • False Positives/Negatives: Like most linters, the tool may produce false positives and false negatives, aiming for a reasonable signal-to-noise ratio.
  • Future Work:
    • Extending Abstract Domain: Continuously refining the abstract datatype system to cover more data science library functions and nuances.
  • Supporting Code Annotations: Allowing users to annotate their code to suppress false positives or provide hints for more precise analysis.
  • Integrating with More Libraries: Expanding support to a wider range of data science libraries beyond pandas and scikit-learn.
  • Experimental Evaluation: Conducting a thorough experimental evaluation on a broad benchmark of Jupyter notebooks (e.g., from Kaggle) to assess the tool's effectiveness.
  • Developing Annotation Hints: Automatically suggesting annotations to users based on the analysis results.

Summary updated

BibTeX
@inproceedings{dolcetti2024towards,
  author = {Greta Dolcetti and Agostino Cortesi and Caterina Urban and Enea Zaffanella},
  title = {{Towards a High Level Linter for Data Science}},
  booktitle = {Nsad@sas},
  year = {2024},
  doi = {10.1145/3689609.3689996},
}
Figure