Abstract
Standard Automated Machine Learning (AutoML) frameworks typically treat the construction of machine learning pipelines as a complex, high-dimensional optimization problem. These systems assume that the selection of components—such as scalers, feature selectors, and classifiers—is deeply interdependent, requiring sophisticated search strategies like Bayesian Optimization or Genetic Algorithms. This paper challenges that assumption by introducing "Naive" and "Quasi-Naive" AutoML. By optimizing pipeline components in isolation or sequential order, these approaches demonstrate that the interactions between components are often weaker than previously thought. Experimental results across 71 datasets show that naive approaches are highly competitive, particularly in time-constrained scenarios, often outperforming state-of-the-art tools in short-run evaluations.
Introduction
Building an effective machine learning pipeline involves a daunting series of decisions: one must select a feature scaler, a feature selector, and a classifier, while simultaneously tuning hyperparameters for each. Current state-of-the-art (SOTA) AutoML frameworks, such as auto-sklearn and GAMA, treat this as a "black-box" optimization problem. They operate on the premise that the search space is multiplicative—that the choice of a specific scaler only yields optimal results when paired with a specific classifier and a specific set of hyperparameters.
This paper asks a provocative question: Is this complexity necessary? By deconstructing the search space and treating component selection as a series of independent or sequential tasks, the authors explore whether near-optimal pipelines can be found without the overhead of joint optimization.
Methodology
The researchers propose two simplified approaches to AutoML that decouple Algorithm Selection from Hyperparameter Optimization (HPO):
- Naive AutoML: This approach assumes total independence between pipeline slots. It iterates through each slot (e.g., the classifier slot), identifies the best-performing algorithm using default parameters, and only then performs a local random search to tune the hyperparameters for those specific selections. This collapses the search space from multiplicative to additive.
- Quasi-Naive AutoML: This version introduces a degree of interaction by optimizing slots in a specific sequence—typically selecting the classifier first and then the pre-processors. When selecting a component for a subsequent slot, the previously chosen "best" components remain fixed. This allows the system to account for some dependencies without the exponential explosion of a full joint search.
Both methods prioritize finding the right algorithm using sensible defaults before investing computational budget into fine-tuning, reflecting the observation that hyperparameter tuning often yields marginal gains compared to selecting the correct model architecture.
Experimental Setup
The effectiveness of these naive strategies was evaluated using a diverse portfolio of 71 classification datasets from the OpenML platform. These datasets represent a wide variety of real-world challenges, including binary and multi-class problems with varying feature densities and instance counts. To ensure a rigorous comparison, the researchers unified the search space and standardized validation mechanisms across all tested optimizers, allowing for a true "head-to-head" performance analysis.
Key Insights and Results
The Strength of Independence
The core finding of the study is that for many real-world datasets, the interactions between pipeline components are relatively weak. Picking the "locally" best component frequently leads to a "globally" excellent pipeline. This "naivety assumption" allows the optimizer to evaluate the most promising candidates much faster than traditional methods.
Performance Benchmarks
- Short-Run Dominance: In one-hour search windows, Quasi-Naive AutoML emerged as the strongest performer. It separated itself from the SOTA group early in the process and maintained a superior median rank across the 71 datasets.
- Long-Run Competitiveness: In extended 24-hour runs, sophisticated tools like
auto-sklearneventually overtook the naive approaches on approximately 75% of the datasets. However, the performance gap was remarkably small, frequently less than 0.02 in AUROC. - Efficiency and Robustness: Quasi-Naive AutoML achieved "de-facto optimal" performance in over 90% of cases, suggesting it serves as an incredibly robust baseline that is difficult to beat significantly even with massive computational resources.
Discussion and Limitations
While the naive approach is surprisingly effective for standard tabular data, the authors note several limitations:
- Deep Learning: The assumption that hyperparameter tuning is secondary to algorithm selection likely fails in the context of neural networks, where performance is highly sensitive to learning rates and architectural parameters.
- Pipeline Topology: The current study focuses on linear pipelines. Extending these naive assumptions to complex directed acyclic graphs (DAGs) or tree-shaped pipelines remains an open challenge.
- Component Compatibility: Independent selection can occasionally result in incompatible components (e.g., a scaler producing negative values for a model requiring non-negative inputs). While the authors utilized a simple "repair" strategy, more robust handling of these constraints is needed.
Conclusion
Naive Automated Machine Learning demonstrates that the "joint optimization" paradigm, while theoretically sound, may be overkill for many practical applications. By prioritizing simplicity and sequential decision-making, Quasi-Naive AutoML provides a high-performance baseline that is faster, more interpretable, and highly competitive with complex SOTA frameworks. This work opens the door for more interactive AutoML systems where human experts can easily intervene in the sequential selection process.
Summary updated