Helix: Holistic Optimization for Accelerating Iterative Machine Learning

Doris Xin, Stephen Macke, Litian Ma, Jialin Liu, Shuchen Song, Aditya G. Parameswaran

Introduction

Machine learning (ML) development is fundamentally a trial-and-error process. Data scientists frequently iterate on their models by tweaking preprocessing steps, adding features, or adjusting hyperparameters. While modern execution engines like Apache Spark or TensorFlow have significantly reduced the time for a single execution, they often fail to optimize the end-to-end iterative cycle. Developers are typically forced to either rerun entire pipelines from scratch—wasting time—or manually manage intermediate results, which is error-prone and inefficient.

HELIX is a holistic optimization system designed to bridge this gap. By treating the entire ML workflow as a directed acyclic graph (DAG) and intelligently managing the reuse and materialization of intermediate results, HELIX can accelerate the cumulative development process by up to 19x.

System Architecture

HELIX is structured into three primary layers: the Programming Interface, the Compilation layer, and the Execution layer.

The HELIX system architecture, illustrating the flow from the Scala DSL to the execution engine.
  1. Programming Interface: Developers define workflows using the Helix Markup Language (HML), a declarative Scala-based DSL. HML covers the entire pipeline, including data preprocessing (DPR), learning/inference (L/I), and post-processing (PPR).
  2. Compilation: The DSL is compiled into a Workflow DAG. HELIX tracks changes across iterations, identifying which parts of the graph are new and which are equivalent to previous runs.
  3. Execution: This layer handles the actual computation, interacting with backends like Spark. It includes a Materialization Optimizer that decides which results are worth saving for future use.

Holistic Optimization Strategy

The core of HELIX lies in solving two distinct optimization problems:

Optimal Execution Plan (OPT-EXEC-PLAN)

When a user modifies a workflow, HELIX must decide which previously materialized results to load and which nodes to recompute. This is not trivial because loading a large dataset from disk might be slower than recomputing it from a smaller, cached ancestor. HELIX models this as a Project Selection Problem, which can be solved optimally in polynomial time using a MAX-FLOW algorithm.

Optimal Materialization Plan (OPT-MAT-PLAN)

Deciding what to save now to benefit future iterations is an NP-Hard problem. HELIX employs a "Streaming OMP" heuristic. It materializes a result if the cost of saving and loading it is significantly lower than the cumulative time required to recompute it and its dependencies.

Iterative Workflow in Practice

To illustrate how HELIX handles iterative changes, consider a Census classification workflow. A developer might add a new feature extractor (msExt) and update the feature list.

A Census workflow example showing the HML code and how HELIX updates the optimized DAG when the code is modified.

As shown in the figure above, HELIX detects the modification and prunes unnecessary branches (like the raceExt node if it's no longer used) while reusing existing results for unchanged nodes. This "program slicing" ensures that only the necessary delta is computed.

Performance and Scalability

The authors evaluated HELIX across diverse real-world tasks, including NLP (Information Extraction), Computer Vision (MNIST), and Genomics. Key findings include:

  • Speedup: HELIX achieved up to a 19x reduction in cumulative runtime over 10 iterations compared to state-of-the-art systems like KeystoneML.
  • Storage Efficiency: By using its materialization heuristic, HELIX often uses 30x less storage than "materialize everything" strategies while maintaining similar performance gains.
  • Scalability: The system scales linearly with dataset size and outperforms existing distributed frameworks as the number of workers increases.

Conclusion and Future Work

HELIX demonstrates that holistic, cross-iteration optimization is a powerful tool for accelerating ML development. By moving beyond single-run optimization and explicitly modeling the load-vs-compute trade-off, HELIX significantly reduces the "human-in-the-loop" latency. Future work aims to integrate non-JVM libraries like PyTorch and explore more advanced change detection techniques to unlock further reuse opportunities.

Summary updated

BibTeX
@article{xin2018helix,
  author = {Doris Xin and Stephen Macke and Litian Ma and Jialin Liu and Shuchen Song and Aditya G. Parameswaran},
  title = {{Helix: Holistic Optimization for Accelerating Iterative Machine Learning}},
  journal = {Proc. VLDB Endow.},
  year = {2018},
  doi = {10.14778/3297753.3297763},
}
Figure