Improvement of Computational Performance of Evolutionary AutoML in a Heterogeneous Environment

Nikolay O. Nikitin, Sergey Teryoshkin, Valerii Pokrovskii, Sergey Pakulin, Denis A. Nasonov

Abstract

Automated Machine Learning (AutoML) significantly reduces the manual effort required to design machine learning pipelines. However, the computational cost of searching for optimal architectures—particularly when using evolutionary algorithms—remains a major bottleneck. This paper presents a modular framework implemented within the FEDOT ecosystem designed to optimize the evaluation of graph-based machine learning pipelines across heterogeneous hardware environments. By integrating granular caching, multi-level parallelization, and hybrid CPU/GPU/remote execution, the proposed approach significantly reduces the time-to-solution while maintaining or improving model quality.

Introduction

The search space for machine learning pipelines is vast, often involving complex directed acyclic graphs (DAGs) where nodes represent data preprocessing steps or modeling algorithms. Evolutionary algorithms (EA) are highly effective at navigating these spaces but require the training and evaluation of thousands of candidate individuals. In modern computing environments, resources are often heterogeneous, comprising a mix of local CPUs, discrete GPUs, and remote cloud clusters. Existing AutoML tools frequently fail to leverage this heterogeneity efficiently, often treating the pipeline as a monolithic entity or failing to reuse computations at a granular level.

This research addresses these inefficiencies by introducing a modular execution layer that optimizes resource utilization and minimizes redundant computations through structural awareness.

Methodology

The proposed optimization strategy is built upon four primary pillars integrated into the open-source FEDOT framework:

1. Granular Node-Level Caching

Traditional AutoML frameworks often cache entire pipelines. However, in an evolutionary search, many "mutated" pipelines share significant structural overlap with their parents. We implement a relational database-backed (SQLite) caching system that operates at the node level. Using recursive structural hashing, the system identifies if a specific sub-graph (e.g., a specific scaler followed by a specific PCA implementation) has been previously trained. If a match is found, the system retrieves the pre-trained state, avoiding redundant computation.

2. Multi-level Parallelization

The framework employs concurrent evaluation of individuals within a population. To prevent "zombie" pipelines—those that are computationally expensive but slow to converge—from bottlenecking the entire generation, we implement strict timeout logic. This ensures that the evolutionary process maintains a steady velocity.

3. Hybrid CPU/GPU Evaluation

Recognizing that different algorithms perform better on different hardware, the framework allows for node-specific hardware dispatching. For instance, data preprocessing and simple linear models can remain on the CPU, while computationally intensive nodes (such as XGBoost or Support Vector Classifiers) are dispatched to the GPU using the RAPIDS (cuML) library. This "compositional heterogeneity" ensures that each part of the pipeline uses the most efficient resource available.

4. Remote Evaluation via Kubernetes

For large-scale optimization tasks, the framework can offload pipeline evaluations to remote clusters. This system manages containerization, resource limits, and result fetching via a REST API, allowing the local environment to scale horizontally into the cloud or dedicated high-performance computing (HPC) clusters.

Experimental Setup

The performance of the framework was evaluated using two primary data sources:

  • OpenML Benchmarks: 12 diverse classification datasets (e.g., adult, bank-marketing, phoneme, and volkert) were used to test real-world applicability.
  • Synthetic Datasets: Using "Blobs" and "Moons" generators, we created datasets with up to 300,000 rows to test scalability and measure the overhead associated with GPU and remote data transfer.

Key Results and Discussion

Caching and Parallelization

Enabling granular caching resulted in an average 14% increase in the number of pipelines evaluated within a fixed time budget. This increased throughput directly correlated with the discovery of higher-quality models. Furthermore, the system demonstrated near-linear speedup when scaling from 1 to 8 parallel processes, confirming the efficiency of the multi-level parallelization strategy.

Hardware Acceleration

GPU acceleration provided the most significant gains on large-scale synthetic datasets. For datasets with 300,000 rows, GPU-backed nodes (cuML) achieved up to a 95% improvement in fitting time compared to CPU-only execution. However, the researchers identified an "overhead threshold": for smaller datasets, the time required to serialize data and transfer it to GPU memory exceeded the actual training time, suggesting that infrastructure-aware scheduling is critical.

Remote Scaling

While remote evaluation via Kubernetes introduced a baseline orchestration overhead of approximately 10-20 seconds, it proved essential for "heavy" tasks. By offloading these tasks, the system prevented local resource exhaustion and allowed for the simultaneous evaluation of complex pipeline structures that would otherwise be impossible on a single workstation.

Conclusion and Future Work

The integration of structural caching and heterogeneous computing significantly enhances the efficiency of evolutionary AutoML. By making the framework "infrastructure-aware," we can drastically reduce the time required to find optimal machine learning solutions.

Future research will focus on:

  • Reducing Network Bottlenecks: Implementing shared cluster file systems (e.g., NFS) to minimize data transfer overhead in remote execution.
  • Speculative Computing: Developing a "speculative mode" where the system predicts and pre-trains promising pipeline structures before they are officially selected by the evolutionary algorithm.
  • Advanced Scheduling: Utilizing sophisticated orchestrators like Volcano for Kubernetes to better manage batch workloads in high-traffic environments.

Summary updated

BibTeX
@inproceedings{nikitin2023improvement,
  author = {Nikolay O. Nikitin and Sergey Teryoshkin and Valerii Pokrovskii and Sergey Pakulin and Denis A. Nasonov},
  title = {{Improvement of Computational Performance of Evolutionary AutoML in a Heterogeneous Environment}},
  booktitle = {CEC},
  year = {2023},
  doi = {10.1109/cec53210.2023.10254012},
}
Figure