Incremental Execution for Iterative Machine Learning Pipelines

28 concepts18 papers1 proposal

Machine learning development is rarely a linear path from data to model. Instead, it is a fundamentally iterative cycle of trial and error. Whether a researcher is tuning hyperparameters, engineering new features, or updating a model with fresh data, the process involves making small changes and observing the results.

However, most traditional execution frameworks treat every run as a "one-and-done" task. When a developer changes a single parameter in a late-stage component, the system often re-executes the entire pipeline from scratch. As Xin et al. observe, this fails to address the cumulative time lost during these minor iterations, leading to significant computational waste.

The Redundancy of Iteration

The intuition behind incremental execution is simple: if the inputs and parameters of a specific step haven't changed, its output shouldn't be recomputed. The scale of potential savings is massive. Lee et al. found that in manual machine learning workflows, 80% of iterations reuse more than half of the operators from previous versions.

This redundancy isn't limited to manual exploration. In production environments, Xin and Miao analyzed thousands of pipelines at Google and found that consecutive runs often share over 60% of their input data. By applying principles like incremental view maintenance—a technique borrowed from database optimization—systems can avoid re-processing these overlapping data segments.

Caching and Dependency Tracking

To capture these savings, systems must track the dependencies between different stages of a pipeline. Early tools like Experiment.perl treated experiments like software builds, using file timestamps and configuration settings to identify which components were affected by a change and skipping those that were not (Koehn).

Modern frameworks have evolved this into sophisticated caching mechanisms:

  • Persistent Storage: For long-running tasks like Large Language Model (LLM) workflows, Patel et al. demonstrate how persisting computational steps to disk allows researchers to recover from interruptions and avoid expensive redundant model calls.
  • Fine-Grained Reuse: Spiegelberg introduces Tuplex, which caches intermediate results during data preparation, allowing developers to iterate rapidly on data cleaning logic without re-running the entire ingestion process.
  • Version Control: Luo et al. propose MLCask, which integrates Git-like branching with data de-duplication. By identifying identical pipeline components across different branches, the system can achieve up to 7.8x faster execution.

Beyond Simple Caching: Holistic Optimization

The challenge of incremental execution is not just what to cache, but whether caching is worth it. Storing every intermediate result can lead to excessive storage overhead. Xin et al. argue that reuse should be treated as a formal optimization problem, where the system weighs the cost of recomputation against the cost of loading data from storage. Their system, Helix, uses program analysis to make these decisions holistically across multiple iterations.

Other researchers focus on identifying "equivalences"—different computational paths that yield the same result. Kontaxakis et al. represent workloads as directed hypergraphs to find execution plans that swap in pre-computed results or cheaper equivalent tasks, significantly reducing the cost of exploratory machine learning.

Expanding the Scope

Incremental techniques are now being applied to specialized domains within the machine learning lifecycle:

  • Automated ML (AutoML): In evolutionary AutoML, where thousands of graph-based models are evaluated, Nikitin et al. show that caching individual components is crucial for making the search process computationally feasible.
  • Formal Verification: When a model is slightly modified (e.g., via quantization), Ugare et al. demonstrate that reusing "specification trees"—proof traces from previous versions—can speed up the verification of the new model by up to 43x.
  • Declarative Systems: Systems like SystemDS integrate these optimizations directly into the compiler, automatically identifying reuse opportunities across the entire data science lifecycle, from cleaning to debugging (Antonov et al.).

By shifting the focus from optimizing a single execution to optimizing the entire iterative process, these frameworks allow researchers to spend less time waiting for computations and more time exploring the configuration space (Garduño et al.).

Key Foundational Work

An Experimental Management System (2010) — contributed 2 concept elements to this direction.
CSE Framework: A UIMA-based Distributed System for Configuration Space Exploration (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_incremental_pipeline_execution_48403338,
  title = {incremental pipeline execution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incremental-pipeline-execution/rdr_48403338a6f742bd94838eccdc631b04},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_incremental_pipeline_execution_48403338_v1,
  title = {incremental pipeline execution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incremental-pipeline-execution/rdr_48403338a6f742bd94838eccdc631b04/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake