pytest-inline: An Inline Testing Tool for Python

Yu Liu, Zachary Thurston, Alan Han, Pengyu Nie, Milos Gligoric, Owolabi LegunsenarXiv 2023 · arXiv

TL;DR

pytest-inline is the first inline testing framework for Python, designed to make statement-level testing seamless and reusable. Built as a plugin for the popular pytest ecosystem, it allows developers to test complex, individual statements (like regular expressions or bitwise operations) directly within the source code with a negligible runtime overhead of just 1.2%.

What Problem Does This Paper Address?

Traditional testing frameworks operate at the level of unit, integration, or system tests. While effective for overall application logic, these approaches are often too coarse-grained to easily test individual, complex program statements.

Developers frequently need to verify single statements that contain intricate logic—such as complex regular expressions, bit manipulations, or Python "one-liners"—or statements buried deeply within private, hard-to-reach functions. Without framework support, engineers resort to ad-hoc methods like "printf debugging" (adding and deleting temporary print statements) or copying code snippets into web-based regex testers. These practices disrupt developer focus, violate encapsulation principles, and produce ephemeral tests that cannot be reused or automated in continuous integration (CI) pipelines.

Methods

The authors introduce pytest-inline, which provides a structured API and runtime environment for inline testing. An inline test consists of providing arbitrary inputs and test oracles to check the immediately preceding statement.

The pytest-inline API consists of three core components:

  1. Declare (itest()): Signals the framework to process the statement and allows configuration (e.g., custom names, timeouts, or tags).
  2. Assign (given()): Binds test inputs to the variables used in the target statement.
  3. Assert (check_*()): Defines the test oracle (e.g., check_eq, check_none) to verify the statement's output.

Architecture

As shown in the architecture diagram below, the framework operates via three internal components:

  • FINDER: Uses Python’s Abstract Syntax Tree (AST) library to locate itest() statements within standard .py files.
  • PARSER: Traverses the AST to pair each inline test with its target statement, extracts arguments, and constructs an independent, encapsulated mini-program.
  • RUNNER: Executes this mini-program in an isolated context containing only the necessary local variables, automatically handling required library imports.
Fig. 2: Architecture of pytest-inline.

The tool builds upon an earlier prototype by introducing advanced features inspired by mature testing frameworks like JUnit, including parameterized testing, parallel execution support via pytest-xdist, and conditional execution using assumptions.

TABLE I: pytest-inline’s features. The top five are in our original prototype [17]; the bottom five are new.

Datasets

Because inline testing is a novel paradigm, open-source benchmarks do not yet natively contain them. To evaluate the tool, the authors utilized a dataset from prior work consisting of 87 inline tests written for 80 target statements across 31 popular open-source Python projects (including prominent repositories like pandas, numpy, black, pipenv, and scrapy).

Key Insights and Ideas

  • Isolated Context Execution: Instead of executing the entire function or file, pytest-inline isolates the target statement and its test logic. This prevents side effects and eliminates the need to mock or set up complex global states just to test a single line of code.
  • Plugin-First Design: By implementing the framework as a pytest plugin rather than a standalone tool, it seamlessly inherits the existing Python testing ecosystem, allowing developers to run inline tests alongside traditional unit tests using familiar command-line arguments.
  • Amortized Discovery Costs: The framework demonstrates that processing overhead decreases significantly when tests are run repeatedly, thanks to AST parsing costs being amortized and Python runtime warm-up effects.

Key Results

The authors evaluated pytest-inline using both standalone and integrated experiments to measure performance and runtime overhead.

  • Standalone Performance: Without test duplication, the average execution time per inline test is a mere 0.094 seconds. When simulating a larger test suite by duplicating tests up to 1,000 times, the average time per test drops to 0.001 seconds due to caching and warm-up efficiencies.
TABLE II: Results of standalone experiments. Dup = dupli- cation count, #IT= total no. of inline tests, TIT[s]= total inline tests run time, tIT[s]= average run time per inline test.
  • Low Integration Overhead: When running inline tests simultaneously with existing unit test suites across 10 large open-source projects, the runtime overhead was found to be negligible at just 0.012x (a 1.2% increase in testing time). Even when inline tests were duplicated 1,000 times to match the volume of unit tests, the overhead remained low at 6.2%.
TABLE III: Results of integrated experiments. Dup = duplica- tion times, #UT= total no. of unit tests, #IT= total no. of inline tests, TUT [s]= total time to run unit tests, TITE [s]= total time to run unit tests with inline tests enabled, OITE= overhead of running unit tests with inline tests enabled, TITD[s]= total time to run unit tests with inline tests disabled, OITD= overhead of running unit tests with inline tests disabled.

Limitations and Future Work

While the performance overhead is currently minimal, the authors note that if developers write thousands of inline tests per project, the cumulative execution time could eventually impact CI/CD pipelines. To mitigate this, future work will focus on developing specialized regression test selection (RTS) techniques tailored specifically for inline tests, ensuring only affected statements are re-run.

Additionally, the authors plan to integrate pytest-inline with mocking frameworks (like pytest-mock) to support statement-level testing for lines of code that depend on external dependencies, such as databases or file systems.

Summary updated

Abstract

We present pytest-inline, the first inline testing framework for Python. We recently proposed inline tests to make it easier to test individual program statements. But, there is no framework-level support for developers to write inline tests in Python. To fill this gap, we design and implement pytest-inline as a plugin for pytest, the most popular Python testing framework. Using pytest-inline, a developer can write an inline test by assigning test inputs to variables in a target statement and specifying the expected test output. Then, pytest-inline runs each inline test and fails if the target statement's output does not match the expected output. In this paper, we describe our design of pytest-inline, the testing features that it provides, and the intended use cases. Our evaluation on inline tests that we wrote for 80 target statements from 31 open-source Python projects shows that using pytest-inline incurs negligible overhead, at 0.012x. pytest-inline is integrated into the pytest-dev organization, and a video demo is at https://www.youtube.com/watch?v=pZgiAxR_uJg.
BibTeX
@misc{liu2023pytest,
  author = {Yu Liu and Zachary Thurston and Alan Han and Pengyu Nie and Milos Gligoric and Owolabi Legunsen},
  title = {{pytest-inline: An Inline Testing Tool for Python}},
  year = {2023},
  eprint = {2305.13486},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13486},
}
Figure