Towards Reproducible Empirical Research in Meta-Learning

Adriano Rivolli, Luís Paulo F. Garcia, Carlos Soares, Joaquin Vanschoren, André C. P. L. F. de Carvalho

Abstract

Meta-learning is increasingly used to support the recommendation of machine learning algorithms and their configurations. Such recommendations are made based on meta-data, consisting of performance evaluations of algorithms on prior datasets, as well as characterizations of these datasets. These characterizations, also called meta-features, describe properties of the data which are predictive for the performance of machine learning algorithms trained on them. Unfortunately, despite being used in a large number of studies, meta-features are not uniformly described, organized and computed, making many empirical studies irreproducible and hard to compare. This paper aims to deal with this by systematizing and standardizing data characterization measures for classification datasets used in meta-learning. Moreover, it presents MFE, a new tool for extracting meta-features from datasets and identifying more subtle reproducibility issues in the literature, proposing guidelines for data characterization that strengthen reproducible empirical research in meta-learning.

Introduction

In machine learning, meta-learning serves as a powerful paradigm for recommending optimal algorithms or configurations for new tasks. This process relies heavily on meta-features—numerical descriptors that capture the essential characteristics or "DNA" of a dataset, such as the ratio of features to instances or class label entropy. However, the field currently lacks a "standard library" for these features. Disparate naming conventions, omitted extraction details (like hyperparameters), and inconsistent summarization functions (e.g., mean vs. maximum) have led to a reproducibility crisis. Without standardized definitions, comparing meta-learning studies becomes nearly impossible.

A Formal Framework for Meta-Features

To address these inconsistencies, the authors propose a formal framework defining a meta-feature $f$ as the combination of a characterization measure $m$ and a summarization function $\sigma$: $$f(D) = \sigma(m(D, h_m), h_s)$$ where $h_m$ and $h_s$ represent hyperparameters for the measure and summary, respectively. This distinction is critical: a measure (like correlation) may produce multiple values, which only become a single "meta-feature" after summarization.

The paper introduces a rigorous taxonomy to classify these measures based on their input requirements and output properties, as shown in the table below.

Taxonomy of data characterization measures based on input and output levels.

Categorization and the MFE Tool

Meta-features are organized into six major groups: Simple (basic stats), Statistical (central tendency/dispersion), Information-theoretic (entropy), Model-based (tree-based features), Landmarking (performance of fast learners), and Others (complex network measures).

To operationalize this taxonomy, the authors introduced MFE (Meta-Feature Extractor), an open-source tool available in Python and R. MFE provides a "white-box" implementation of over 100 measures with sensible defaults, aiming to replace the "black-box" tools of the past. The following table illustrates how specific simple measures are mapped to the proposed taxonomy within the tool.

Examples of simple meta-features and their properties within the proposed taxonomy.

Key Insights for Reproducibility

The study highlights several often-overlooked factors in meta-feature extraction that affect reproducibility:

  • The Measure vs. Feature Distinction: The choice of summary function is a hidden hyperparameter that significantly impacts results.
  • Pre-processing Bias: Decisions on how to handle categorical data for numerical measures (e.g., binarization) are rarely reported but can fundamentally alter the meta-learning model's performance.
  • Non-Determinism: Model-based and landmarking features can vary between runs due to random sampling, necessitating fixed seeds for reproducibility.

Conclusion and Future Work

While this work standardizes characterization for classification, several avenues for future research remain. Currently, MFE is limited to classification tasks; extending it to regression and clustering is essential. Additionally, the tool requires manual imputation for missing values, and the sensitivity of meta-learning models to specific hyperparameter choices in feature extraction warrants further large-scale empirical investigation. By providing a unified taxonomy and the MFE tool, this research lays the groundwork for more transparent and comparable meta-learning research.

Summary updated

Abstract

Meta-learning is increasingly used to support the recommendation of machine learning algorithms and their configurations. Such recommendations are made based on meta-data, consisting of performance evaluations of algorithms on prior datasets, as well as characterizations of these datasets. These characterizations, also called meta-features, describe properties of the data which are predictive for the performance of machine learning algorithms trained on them. Unfortunately, despite being used in a large number of studies, meta-features are not uniformly described, organized and computed, making many empirical studies irreproducible and hard to compare. This paper aims to deal with this by systematizing and standardizing data characterization measures for classification datasets used in meta-learning. Moreover, it presents MFE, a new tool for extracting meta-features from datasets and identifying more subtle reproducibility issues in the literature, proposing guidelines for data characterization that strengthen reproducible empirical research in meta-learning.
BibTeX
@misc{rivolli2018towards,
  author = {Adriano Rivolli and Luís Paulo F. Garcia and Carlos Soares and Joaquin Vanschoren and André C. P. L. F. de Carvalho},
  title = {{Towards Reproducible Empirical Research in Meta-Learning}},
  year = {2018},
  eprint = {1808.10406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.10406},
}

Loading related content...

Figure