Learning from Metadata in Repositories

Pavel Brazdil, Jan N. van Rijn, Carlos Soares, Joaquin VanschorenCognitive 2022 · DOI · OpenAlex

Abstract

This chapter describes the various types of experiments that can be done with the vast amount of data stored in experiment databases. We focus on three types of experiments conducted with the data stored in OpenML, demonstrating how to extract high-level "meta-knowledge" to understand algorithm behavior, hyperparameter importance, and model diversity.

Introduction

Traditional machine learning research often suffers from "locality," where algorithms are evaluated on a limited set of datasets. This narrow focus fails to capture the broader landscape of algorithm behavior, making it difficult to discern whether success is due to dataset-specific quirks, fortunate hyperparameter settings, or fundamental architectural advantages. By leveraging experiment databases like OpenML, which store full metadata including hyperparameters and per-instance predictions, researchers can transform "big data of experiments" into actionable insights for model selection and automated machine learning (AutoML).

Methodology

The meta-analysis is categorized into three distinct levels:

  1. Model-level: Analyzing performance variance relative to hyperparameters using techniques like functional ANOVA.
  2. Data-level: Investigating correlations between dataset meta-features (e.g., instance count, dimensionality) and algorithm success.
  3. Method-level: Determining algorithm similarity through Classifier Output Difference (COD) and Hierarchical Agglomerative Clustering (HAC).

Robust statistical evaluation is central to this approach. Rather than relying solely on mean accuracy, the authors utilize distribution analysis and significance testing to establish reliable rankings across diverse tasks.

Violin plots showing the distribution of Area Under ROC Curve across multiple datasets for various classification algorithms.

Experimental Setup

The study primarily utilizes the OpenML ecosystem, focusing on:

  • OpenML-CC18: A curated benchmark suite of 45 datasets for clustering algorithm behaviors.
  • 105-dataset collection: A large-scale ranking of over 30 popular classifiers, including Random Forests and SVMs.
  • UCI "Letter" dataset: A specific case study for analyzing hyperparameter sensitivity.

Analysis of Algorithm Behavior

The "Tuning Tax" and Hyperparameter Sensitivity

Algorithms exhibit varying degrees of "tunability." Support Vector Machines (SVMs), for instance, are highly sensitive to hyperparameters like gamma, with performance swinging significantly based on the configuration. In contrast, Random Forests and Decision Trees are remarkably robust, often achieving near-optimal performance with default settings. This discrepancy defines the "tuning tax"—the computational cost required to reach an algorithm's peak performance.

Predictive accuracy as a function of hyperparameter values across different datasets, illustrating varying sensitivity to tuning.

Model Selection Trade-offs

Analysis reveals that non-linear models typically gain a statistically valid advantage over linear models only when the number of observations is high. In many other regimes, simpler linear models are statistically equivalent while offering superior speed and interpretability. Furthermore, for ensemble building, diversity often trumps individual accuracy; selecting "distantly clustered" models in the prediction space (using COD metrics) can lead to more effective ensembles than simply picking the top-performing but highly correlated models.

Results

  • Consistent Performers: Random Forest and Logistic Model Trees (LMT) emerged as the most consistently high-ranking algorithms across 105 datasets when using default parameters.
  • Optimization Impact: Hyperparameter tuning significantly improved SVM performance across nearly all datasets, while the gains for Decision Trees were often negligible.
  • Statistical Equivalence: Nemenyi tests identified large "cliques" of statistically indistinguishable algorithms, suggesting that the choice between high-performing ensemble methods may be less critical than proper data preprocessing.

Conclusion and Future Work

While these meta-analyses provide a comprehensive snapshot, they are "living" documents that must evolve as new algorithms emerge. Current findings highlight the potential for Meta-Learning, where insights from experiment databases are used to train "tuner" models. These models can analyze a new dataset's meta-features to instantly recommend optimal algorithms and hyperparameter ranges, significantly reducing the computational cost of model selection and bypassing exhaustive grid searches.

Summary updated

Abstract

Abstract This chapter describes the various types of experiments that can be done with the vast amount of data, stored in experiment databases. We focus on three types of experiments done with the data stored in OpenML.
BibTeX
@inproceedings{brazdil2022learning,
  author = {Pavel Brazdil and Jan N. van Rijn and Carlos Soares and Joaquin Vanschoren},
  title = {{Learning from Metadata in Repositories}},
  booktitle = {Cognitive},
  year = {2022},
  doi = {10.1007/978-3-030-67024-5\_17},
}
Figure