Abstract
Automated Machine Learning (AutoML) has emerged as a vital field for streamlining the development of machine learning pipelines. However, despite its practical success, the field often relies on ad-hoc heuristics for search and resource allocation. This paper provides a theoretical bridge between AutoML and classical AI theories of bounded rationality and rational metareasoning. By framing AutoML tools as resource-constrained agents that must optimize their own computational processes, we provide a foundation for transitioning AutoML from heuristic-driven search to mathematically grounded, optimal resource allocation.
Introduction
In standard supervised machine learning, the primary objective is to identify the most accurate model for a specific dataset. This pursuit often assumes an idealized environment where the computational cost of the search process itself is a secondary concern. In practice, however, Automated Machine Learning (AutoML) systems—which automate algorithm selection, pipeline construction, and hyperparameter tuning—operate within a massive and complex search space.
Because evaluating a single candidate pipeline can be computationally expensive, AutoML systems must function under strict resource constraints, such as time limits or fixed compute budgets. Currently, many AutoML tools utilize various heuristics to navigate these constraints, yet they lack a unifying theoretical framework. This paper addresses this gap by mapping AutoML to Herbert Simon’s concept of "bounded rationality" and the framework of "rational metareasoning."
Theoretical Framework: Mapping Rationality to AutoML
This conceptual analysis re-examines AutoML by aligning it with four distinct types of rationality defined in classical AI literature:
- Perfect and Calculative Rationality: This represents the traditional ML approach, which seeks the absolute best action or model regardless of the computational cost required to find it. While theoretically ideal, it is practically infeasible in the vast search spaces of AutoML.
- Meta-Level Rationality (Rational Metareasoning): This approach bifurcates the problem into an object-level (the act of fitting a specific model to data) and a meta-level (the decision-making process regarding which model to evaluate next). The core of this technique is calculating the "value of computation," which involves weighing the expected improvement a candidate model might bring against the cost of its evaluation.
- Bounded Optimality: This focuses on optimizing the actual program running on a specific machine. It acknowledges that an agent’s performance is limited by its architecture and the time available, seeking the best possible performance within those specific bounds.
By viewing existing AutoML tools like auto-sklearn and ML-Plan through this lens, it becomes clear that their search mechanisms are essentially early, implicit forms of metareasoning. For example, auto-sklearn’s use of "warm starting" (prioritizing historically successful models) is a practical application of meta-level reasoning to reduce search costs.
Key Insights: AutoML as a Meta-Level Task
Bounded Rationality and Satisficing
AutoML is inherently a problem of bounded rationality. Because the space of all possible machine learning pipelines is too vast for exhaustive exploration, an AutoML agent cannot achieve perfect rationality. Instead, it must "satisfice"—a term coined by Herbert Simon—meaning it must find a solution that is "good enough" within the provided constraints.
Object-Level vs. Meta-Level Reasoning
A critical distinction introduced in this paper is that standard machine learning is an object-level task (an agent mapping data to a hypothesis), whereas AutoML is a meta-level task. In AutoML, the agent is reasoning about which object-level agent (model/pipeline) to deploy. This shift in perspective allows us to apply Bayesian decision theory to the search process itself.
Data as a Bounded Resource
Resource constraints in AutoML extend beyond CPU cycles and time; data itself is a bounded resource. Deciding how to partition data into training and validation sets to evaluate a candidate model efficiently is a classic metareasoning problem. Optimal data allocation can significantly accelerate the search process by providing "just enough" evidence to discard poor candidates early.
Implications for Future Research
The primary contribution of this work is the theoretical alignment of AutoML with classical AI decision theory. The authors suggest that the "Aha!" moment for the field lies in the explicit design of AutoML systems using the formal mathematics of rational metareasoning.
Currently, many AutoML features—such as the division of search and selection phases—are implemented as ad-hoc heuristics. By adopting a formal metareasoning approach, future systems could move toward provably optimal choices regarding how to spend each unit of compute time. This would transition the field from "guessing" the next best step to a rigorous calculation of the expected value of the next computation.
Conclusion and Future Work
While this paper establishes the philosophical and conceptual bridge between AutoML and bounded rationality, it represents only the beginning of this integration. Future work is required to provide rigorous mathematical formalizations and to develop new AutoML systems built explicitly on these principles. The authors propose a closer inspection of sub-fields such as algorithm selection and dynamic algorithm configuration to determine the extent to which current methods already act as bounded rational agents. Ultimately, adopting metareasoning techniques promises to not only improve our understanding of existing tools but to fundamentally enhance their efficiency and performance under real-world resource constraints.
Summary updated