GAMA: A General Automated Machine Learning Assistant

Pieter Gijsbers, Joaquin Vanschoren

TL;DR

GAMA (General Automated Machine learning Assistant) is a modular, open-source AutoML tool designed to move beyond "black-box" automation. It allows users to easily swap search algorithms and post-processing steps while providing a dedicated graphical interface to visualize and analyze the optimization process in real-time.

What Problem Does This Paper Address?

Automated Machine Learning (AutoML) has become a powerful way to find the best machine learning pipelines (preprocessing + model selection + hyperparameter tuning) without manual trial and error. However, most existing tools like auto-sklearn or TPOT are monolithic. They often operate as "black boxes" where the user has little control over the internal search logic or visibility into why certain pipelines were chosen.

This lack of transparency and modularity creates a barrier for two groups:

  1. Researchers: Who need to isolate and test specific components (like a new search algorithm) without rebuilding an entire AutoML framework.
  2. Practitioners: Who want to understand the trade-offs the system is making or fine-tune the automation process for specific constraints.

GAMA addresses this by providing a "pluggable" architecture where the search strategy, post-processing, and evaluation are decoupled.

Methods

GAMA’s core innovation is its Modular AutoML Pipeline. Instead of a fixed routine, it allows users to compose a system from three main stages:

  • Search Algorithms: GAMA currently supports three distinct flavors:
  • Random Search: The baseline approach.
  • ASHA (Asynchronous Successive Halving): A resource-efficient method that uses "low-fidelity" estimates to quickly kill off underperforming pipelines.
  • Asynchronous Evolutionary Algorithm: Based on NSGA-II, this uses "survival of the fittest" to evolve pipelines. Crucially, GAMA implements this asynchronously, meaning it doesn't waste time waiting for a full "generation" to finish before starting new evaluations.
  • Post-processing: Once the search is done, GAMA can either:
  • Pick the Single Best pipeline discovered.
  • Build an Ensemble (using the Caruana technique) that combines multiple strong pipelines to improve robustness.
  • Transparency & Interface: GAMA includes a web-based GUI that logs the creation and evaluation of every pipeline. It visualizes convergence rates and, for evolutionary search, tracks "parent-child" relationships between models.

Datasets

The authors evaluated GAMA using the open-source AutoML Benchmark, comparing it against industry standards. The evaluation covered several large and challenging datasets, including:

  • Binary Classification: numeral28.6, higgs, miniboone, and airlines.
  • Multi-class Classification: helena, connect-4, jannis, and covertype.

Key Insights and Ideas

  • Ease of Extension: One of the most impressive claims is that new search algorithms can be integrated into GAMA with fewer than 170 lines of code. This is possible because GAMA provides shared utilities for logging, parallelization, and resource management.
  • Asynchronicity over Generations: By moving away from synchronous "generations" in evolutionary search, GAMA maximizes CPU/GPU utilization, as workers never sit idle waiting for a slow-training model to finish.
  • User-Centric Design: By mimicking the scikit-learn API (fit, predict), GAMA remains accessible to beginners while the modular backend caters to power users.

Key Results

The benchmarking results (visualized in the paper's Figure 2) show that GAMA is highly competitive with established frameworks:

  • Performance Parity: In terms of AUC (Area Under the Curve) and Logloss, GAMA consistently performs within the same clusters as auto-sklearn, TPOT, and H2O AutoML.
  • Flexibility without Penalty: The results demonstrate that GAMA’s modularity does not come at the cost of predictive accuracy. It achieves "state-of-the-art" levels of performance while offering significantly more architectural flexibility than its peers.

Limitations and Future Work

While GAMA is designed for extensibility, its current "out-of-the-box" library is still growing:

  • Algorithm Variety: It currently features only three search algorithms and two post-processing steps. The authors plan to expand this library significantly.
  • Cold Start: The current version does not yet utilize "warm-starting" (using meta-data from previous tasks to jumpstart the search on a new dataset), which is a planned feature for future releases.
  • Pipeline Complexity: Future work aims to extend the AutoML pipeline with additional steps, such as more complex meta-learning components.

Summary updated

Abstract

The General Automated Machine learning Assistant (GAMA) is a modular AutoML system developed to empower users to track and control how AutoML algorithms search for optimal machine learning pipelines, and facilitate AutoML research itself. In contrast to current, often black-box systems, GAMA allows users to plug in different AutoML and post-processing techniques, logs and visualizes the search process, and supports easy benchmarking. It currently features three AutoML search algorithms, two model post-processing steps, and is designed to allow for more components to be added.
BibTeX
@inproceedings{gijsbers2020gama,
  author = {Pieter Gijsbers and Joaquin Vanschoren},
  title = {{GAMA: A General Automated Machine Learning Assistant}},
  booktitle = {ECML/PKDD},
  year = {2020},
  doi = {10.1007/978-3-030-67670-4\_39},
}

Loading related content...

Figure