Abstract
This chapter describes the various types of experiments that can be done with the vast amount of data stored in experiment databases. We focus on three types of experiments conducted with the data stored in OpenML, demonstrating how to extract high-level "meta-knowledge" to understand algorithm behavior, hyperparameter importance, and model diversity.
Introduction
Traditional machine learning research often suffers from "locality," where algorithms are evaluated on a limited set of datasets. This narrow focus fails to capture the broader landscape of algorithm behavior, making it difficult to discern whether success is due to dataset-specific quirks, fortunate hyperparameter settings, or fundamental architectural advantages. By leveraging experiment databases like OpenML, which store full metadata including hyperparameters and per-instance predictions, researchers can transform "big data of experiments" into actionable insights for model selection and automated machine learning (AutoML).
Methodology
The meta-analysis is categorized into three distinct levels:
- Model-level: Analyzing performance variance relative to hyperparameters using techniques like functional ANOVA.
- Data-level: Investigating correlations between dataset meta-features (e.g., instance count, dimensionality) and algorithm success.
- Method-level: Determining algorithm similarity through Classifier Output Difference (COD) and Hierarchical Agglomerative Clustering (HAC).
Robust statistical evaluation is central to this approach. Rather than relying solely on mean accuracy, the authors utilize distribution analysis and significance testing to establish reliable rankings across diverse tasks.
Experimental Setup
The study primarily utilizes the OpenML ecosystem, focusing on:
- OpenML-CC18: A curated benchmark suite of 45 datasets for clustering algorithm behaviors.
- 105-dataset collection: A large-scale ranking of over 30 popular classifiers, including Random Forests and SVMs.
- UCI "Letter" dataset: A specific case study for analyzing hyperparameter sensitivity.
Analysis of Algorithm Behavior
The "Tuning Tax" and Hyperparameter Sensitivity
Algorithms exhibit varying degrees of "tunability." Support Vector Machines (SVMs), for instance, are highly sensitive to hyperparameters like gamma, with performance swinging significantly based on the configuration. In contrast, Random Forests and Decision Trees are remarkably robust, often achieving near-optimal performance with default settings. This discrepancy defines the "tuning tax"—the computational cost required to reach an algorithm's peak performance.
Model Selection Trade-offs
Analysis reveals that non-linear models typically gain a statistically valid advantage over linear models only when the number of observations is high. In many other regimes, simpler linear models are statistically equivalent while offering superior speed and interpretability. Furthermore, for ensemble building, diversity often trumps individual accuracy; selecting "distantly clustered" models in the prediction space (using COD metrics) can lead to more effective ensembles than simply picking the top-performing but highly correlated models.
Results
- Consistent Performers: Random Forest and Logistic Model Trees (LMT) emerged as the most consistently high-ranking algorithms across 105 datasets when using default parameters.
- Optimization Impact: Hyperparameter tuning significantly improved SVM performance across nearly all datasets, while the gains for Decision Trees were often negligible.
- Statistical Equivalence: Nemenyi tests identified large "cliques" of statistically indistinguishable algorithms, suggesting that the choice between high-performing ensemble methods may be less critical than proper data preprocessing.
Conclusion and Future Work
While these meta-analyses provide a comprehensive snapshot, they are "living" documents that must evolve as new algorithms emerge. Current findings highlight the potential for Meta-Learning, where insights from experiment databases are used to train "tuner" models. These models can analyze a new dataset's meta-features to instantly recommend optimal algorithms and hyperparameter ranges, significantly reducing the computational cost of model selection and bypassing exhaustive grid searches.
Summary updated