Hands-On Automated Machine Learning Tools: Auto-Sklearn and Auto-PyTorch

Marius LindauerAMIR@ECIR 2019 · OpenReview

Abstract

Achieving state-of-the-art performance in machine learning typically requires extensive expertise to navigate the complex landscape of algorithm selection and hyperparameter optimization. This paper provides an overview of two prominent Automated Machine Learning (AutoML) frameworks—auto-sklearn and Auto-PyTorch—presented at the 1st Interdisciplinary Workshop on Algorithm Selection and Meta-Learning in Information Retrieval (AMIR 2019). These tools aim to democratize access to high-performance machine learning by automating the "expert bottleneck" of model selection and tuning.

Introduction

Machine learning has become a cornerstone of modern data science, yet achieving peak performance is rarely a "plug-and-play" process. For many practitioners, the transition from a raw dataset to a high-performing model involves navigating a massive search space: choosing between disparate algorithms (e.g., Random Forests vs. Support Vector Machines) and then meticulously tuning their associated hyperparameters.

This manual process creates a significant "expert bottleneck" characterized by two primary issues:

  1. Time Consumption: Manual tuning is iterative, slow, and computationally expensive.
  2. Accessibility: High-level performance is often locked behind years of specialized knowledge, making it difficult for non-experts to apply machine learning effectively to their specific domain problems.

To address these challenges, Marius Lindauer and his team introduced auto-sklearn and Auto-PyTorch, tools designed to provide "out-of-the-box" state-of-the-art performance for both classical machine learning and deep learning.

Automated Frameworks

The development of these tools recognizes that "classical" machine learning and deep learning require fundamentally different optimization strategies. Consequently, two specialized frameworks were developed rather than a single monolithic system.

auto-sklearn

Designed for classical machine learning tasks, auto-sklearn builds upon the popular scikit-learn library. It automates the entire machine learning pipeline, which includes:

  • Data Preprocessing: Handling missing values and categorical encoding.
  • Feature Preprocessing: Scaling, normalization, and dimensionality reduction.
  • Algorithm Selection: Choosing the most appropriate classifier or regressor.
  • Hyperparameter Tuning: Optimizing the settings of the selected algorithm to maximize performance.

Auto-PyTorch

As deep learning became more prevalent, the need for specialized automation within the PyTorch ecosystem grew. Auto-PyTorch focuses on deep learning workflows by automating:

  • Neural Architecture Search (NAS): Finding the optimal structure for a neural network.
  • Hyperparameter Optimization: Tuning learning rates, batch sizes, and optimizer settings specifically for deep learning models.

The Philosophy of Democratization

The core philosophy behind these tools is the democratization of state-of-the-art (SOTA) performance. By automating the "black magic" of hyperparameter tuning and architecture search, these frameworks lower the barrier to entry for non-experts.

The primary value proposition is the drastic reduction of human time spent on trial-and-error. By shifting the burden of optimization from the researcher to the software, practitioners can focus their efforts on higher-level tasks such as problem definition, data quality, and the interpretation of results rather than manual tuning.

Conclusion

Auto-sklearn and Auto-PyTorch represent significant steps toward making high-performance machine learning accessible to a broader audience. By providing automated, specialized tools for both classical and deep learning, these frameworks allow users to achieve competitive results with minimal configuration. As AutoML continues to evolve, the focus remains on increasing efficiency and ensuring that the power of state-of-the-art machine learning is available to all researchers, regardless of their level of expertise.

Summary updated

BibTeX
@inproceedings{lindauer2019hands,
  author = {Marius Lindauer},
  title = {{Hands-On Automated Machine Learning Tools: Auto-Sklearn and Auto-PyTorch}},
  booktitle = {AMIR@ECIR},
  year = {2019},
  url = {https://openreview.net/forum?id=7g2bahBggyP},
}
Figure