Training and Cross-Validating Machine Learning Pipelines with Limited Memory

Martin Hirzel, Kiran Kate, Louis Mandel, Avraham ShinnarAutoML 2024 · OpenReview

TL;DR

This paper introduces a novel approach using "monoids" and "task graphs" to train and cross-validate complex machine learning pipelines on large datasets using limited memory. This allows for efficient AutoML on modest hardware, democratizing access to powerful ML capabilities.

What Problem Does This Paper Address?

Automated Machine Learning (AutoML) aims to automate the process of finding optimal machine learning pipelines. However, two major challenges hinder its widespread adoption: overfitting and excessive resource consumption. Traditional solutions often create a trade-off: cross-validation helps prevent overfitting but demands more computational resources, while optimizing for resources (e.g., by preprocessing data separately) can lead to increased overfitting. This paper tackles the problem of training and cross-validating entire ML pipelines on a single, moderately-sized machine with limited memory, avoiding the need for expensive clusters or powerful hardware.

Methods

The core innovation lies in the use of monoids. A monoid is a mathematical structure with an associative binary operation and an identity element. The authors show how to express the training process of many common machine learning operators (like scalers and imputers) as monoids. This allows data to be processed in batches, with intermediate results (monoids) being combined associatively.

These monoidal operators are then integrated into a task graph framework. Task graphs represent the pipeline as a directed acyclic graph of computational tasks. This framework enables flexible scheduling and resource management, including intelligent spilling of data to disk when memory is scarce. The approach is designed to be compatible with the widely-used sklearn API, making it easy for existing AutoML systems and users to adopt. The authors implement this framework with two backends: pandas for in-memory operations and Spark SQL for distributed processing.

Datasets

The paper evaluates its approach on several datasets, including:

  • KDDCup99
  • steam_reviews
  • ecommerce_2019_oct
  • data_cityofchicago_taxi
  • OpenML datasets (used for RQ5 and RQ6, including spectf, breast-cancer, australian, blood-transfusion, diabetes, credit-g, car, mfeat-factors, kc1, kr-vs-kp, sylvine, phoneme, jungle-chess).

The paper does not introduce new benchmarks but rather uses existing ones to demonstrate the effectiveness of their approach.

Key Insights and Ideas

  • Monoids as a Universal Abstraction for Batching: The paper demonstrates that the mathematical concept of monoids provides a powerful and flexible way to handle batch processing for a wide range of machine learning operators, enabling efficient training with limited memory.
  • Task Graphs for Flexible Pipeline Execution: By representing pipelines as task graphs, the system gains the ability to dynamically schedule computations, manage resources efficiently, and support various training and cross-validation regimes.
  • Bridging the Gap for AutoML on Modest Hardware: The core contribution is enabling complex AutoML tasks on standard, inexpensive compute infrastructure, making advanced ML more accessible.
  • Sklearn API Compatibility: The design choice to adhere to the sklearn API is crucial for practical adoption, allowing seamless integration with existing tools and workflows.

Key Results

  • Memory Efficiency: The batched approach successfully trained pipelines on datasets that would cause standard sklearn pipelines to run out of memory, even on a VM with only 8GB of physical memory. For instance, on the ecommerce_2019_oct dataset, the batched approach worked with a 16GB memory limit, while sklearn failed.
  • Performance: While not always faster, the dual-backend approach (pandas and Spark SQL) offers flexibility. In some cases, Spark SQL provided a speedup over pandas, and vice-versa, allowing users to choose the optimal backend for their specific needs.
  • Accuracy Equivalence: The paper shows that their monoidal implementations of sklearn operators produce results that are nearly identical (differences < 10-7) to the original sklearn implementations.
  • Cross-Validation Regimes: The paper introduces and evaluates four distinct cross-validation regimes, demonstrating their ability to achieve similar results to sklearn's cross_val_score while managing memory constraints.

Limitations and Future Work

  • Non-Monoidal Operators: The paper acknowledges that not all machine learning operators are monoidal. While they discuss "incremental" and "convergent" operators as alternatives, the core monoidal framework is limited to associative operators.
  • Partial Transform Restrictions: The "partial-transform" regime, while enabling incremental training, is more restrictive and requires convergent operators, potentially offering weaker guarantees than the "full-transform" regime.
  • Theoretical Proofs: Due to page limits, the paper prioritizes clear intuitions and illustrative examples over complete formal proofs for theoretical results.
  • Future Work: The paper suggests exploring how to express more complex operators as monoids and further optimizing the task graph scheduling for even greater efficiency. They also mention experiments that leverage convergence, hinting at future directions.

Summary updated

Abstract

While automated machine learning (AutoML) can save human labor in finding well-performing pipelines, it often suffers from two problems: overfitting and using excessive resources. Unfortunately, the solutions are often at odds: cross-validation helps reduce overfitting at the expense of more resources; conversely, preprocessing on a separate compute cluster and then cross-validating only the final predictor saves resources at the expense of more overfitting. This paper shows how to train and cross-validate entire pipelines on a single moderate machine with limited memory by using monoids, which are associative, thus providing a flexible way for handling large data one batch at a time. To facilitate AutoML, our approach is designed to support the common sklearn APIs used by many AutoML systems for pipelines, training, cross-validation, and several operators. Abstracted behind those APIs, our approach uses task graphs to extend the benefits of monoids from operators to pipelines, and provides a multi-backend implementation. Overall, our approach lets users train and cross-validate pipelines on simple and inexpensive compute infrastructure.
BibTeX
@inproceedings{hirzel2024training,
  author = {Martin Hirzel and Kiran Kate and Louis Mandel and Avraham Shinnar},
  title = {{Training and Cross-Validating Machine Learning Pipelines with Limited Memory}},
  booktitle = {AutoML},
  year = {2024},
  url = {https://openreview.net/forum?id=4LkaPSHUQQ},
}
Figure