A Very Brief and Critical Discussion on AutoML

Bin LiuarXiv 2018 · OpenReview

Abstract

This contribution presents a very brief and critical discussion on automated machine learning (AutoML), which is categorized here into two classes, referred to as narrow AutoML and generalized AutoML, respectively. The conclusions yielded from this discussion can be summarized as follows: (1) most existent research on AutoML belongs to the class of narrow AutoML; (2) advances in narrow AutoML are mainly motivated by commercial needs, while any possible benefit obtained is definitely at a cost of increase in computing burdens; (3) the concept of generalized AutoML has a strong tie in spirit with artificial general intelligence (AGI), also called "strong AI", for which obstacles abound for obtaining pivotal progresses.

Introduction

Automated Machine Learning (AutoML) is frequently marketed as a "silver bullet" for the modern data era—a solution designed to remove the human expert from the machine learning loop. By automating data preprocessing, feature engineering, and hyperparameter optimization, AutoML tools promise to democratize AI, allowing non-experts to build high-quality models without deep domain specialization. However, this paper argues that the current state of the art is far from achieving true autonomy. Instead, we are operating within a paradigm of "Narrow AutoML" that merely shifts the burden of expertise while significantly increasing computational costs.

A New Taxonomy: Narrow vs. Generalized AutoML

To understand the limitations of current technology, we must distinguish between two conceptual frameworks:

  1. Narrow AutoML: This describes almost all current research and tools (such as Auto-WEKA, auto-sklearn, or TPOT). These systems use an "outer-loop" algorithm—such as Bayesian Optimization or Genetic Programming—to tune an "inner-loop" machine learning model. While they automate specific tasks, they operate within a constrained, predefined search space.
  2. Generalized AutoML: This represents the theoretical "holy grail"—a system that requires zero expert knowledge or human involvement to perform a machine learning task from start to finish. This level of automation would require the system to understand context and data nuances without guidance.

The "Outermost Layer" Problem

A central critique of Narrow AutoML is the persistent requirement for human intervention at higher levels of abstraction. While a system might use Bayesian Optimization (BO) to pick hyperparameters for a neural network, the BO algorithm itself has hyperparameters—such as kernel types or acquisition functions—that a human expert must still choose.

Consequently, the human expert is not removed from the loop; their role is simply shifted. Instead of tuning a Random Forest directly, the expert is now tuning the AutoML system that tunes the Random Forest. The "outermost layer" of any current system still terminates at a human decision-maker, meaning the expertise requirement is relocated rather than eliminated.

The Compute-Human Tradeoff

There is a significant, often overlooked "compute tax" associated with Narrow AutoML. The reduction in human effort is typically paid for with increased computational burdens. For instance, an experienced data scientist might use their intuition to select a high-performing model architecture in minutes. In contrast, an AutoML tool might require ten hours of GPU-intensive brute-force searching to arrive at the same result.

From a critical standpoint, this is not a fundamental "solution" to the complexity of machine learning; it is a trade-off where human hours are exchanged for electricity and hardware cycles. While commercially attractive for scaling, it does not represent a leap in algorithmic intelligence.

AutoML as a Proxy for AGI

The paper posits that Generalized AutoML is not merely an incremental improvement over current tools. Rather, it is fundamentally a "Strong AI" or Artificial General Intelligence (AGI) problem. Because a truly autonomous system must navigate the "outermost layer" without human guidance, it faces the same fundamental obstacles as AGI. Until breakthroughs occur in the broader field of AGI, Generalized AutoML will likely remain a philosophical goal rather than a technical reality.

Conclusion

While AutoML tools provide valuable utility in streamlining certain workflows, the "automated" label should be approached with caution. Most existing research remains strictly within the realm of Narrow AutoML, driven by commercial demand but limited by the persistent need for human oversight and massive computational resources. Practitioners should remain mindful of the trade-offs between human expertise and the computational costs inherent in current automated pipelines.

Summary updated

Abstract

This contribution presents a very brief and critical discussion on automated machine learning (AutoML), which is categorized here into two classes, referred to as narrow AutoML and generalized AutoML, respectively. The conclusions yielded from this discussion can be summarized as follows: (1) most existent research on AutoML belongs to the class of narrow AutoML; (2) advances in narrow AutoML are mainly motivated by commercial needs, while any possible benefit obtained is definitely at a cost of increase in computing burdens; (3)the concept of generalized AutoML has a strong tie in spirit with artificial general intelligence (AGI), also called "strong AI", for which obstacles abound for obtaining pivotal progresses.
BibTeX
@misc{liu2018very,
  author = {Bin Liu},
  title = {{A Very Brief and Critical Discussion on AutoML}},
  year = {2018},
  eprint = {1811.03822},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.03822},
}
Figure