Metadata Repositories

Pavel Brazdil, Jan N. van Rijn, Carlos Soares, Joaquin VanschorenCognitive 2022 · DOI · OpenAlex

Abstract: This chapter presents a review of online repositories where researchers can share data, code, and experiments. In particular, it covers OpenML, an online platform for sharing and organizing machine learning data automatically. OpenML contains thousands of datasets and algorithms, and millions of experimental results. We describe the basic philosophy involved, and its basic components: datasets, tasks, flows, setups, runs, and benchmark suites. OpenML has API bindings in various programming languages, making it easy for users to interact with the API in their native language. One important feature of OpenML is the integration into various machine learning toolboxes, such as Scikit-learn, Weka, and mlR. Users of these toolboxes can automatically upload all their results, leading to a large repository of experimental results.

Introduction: The Reproducibility Crisis

The machine learning community faces a significant reproducibility crisis. While researchers value data, the actual outputs of experiments—specific hyperparameter settings, exact training splits, and implementation details—are often undocumented or lost in "one-off" studies. This lack of transparency makes it nearly impossible for others to correctly interpret literature or build upon prior work without starting from scratch.

Furthermore, without structured metadata, large-scale meta-analysis remains out of reach. To build systems that automatically choose the best algorithm for a new dataset (metalearning), we require a massive, organized repository of how various algorithms performed on thousands of previous tasks.

The OpenML Architecture

OpenML is an online platform designed to automate the collection of machine learning metadata. Unlike traditional repositories that merely host files, OpenML organizes information into a relational hierarchy that captures the entire lifecycle of an experiment:

  • Datasets: Raw data accompanied by automatically calculated "metafeatures."
  • Tasks: Specific scientific goals defined on a dataset (e.g., 10-fold cross-validation to predict a specific class).
  • Flows: Implementations of algorithms or pipelines (e.g., a specific version of a Random Forest in Scikit-learn).
  • Setups: A Flow combined with specific hyperparameter configurations.
  • Runs: The actual execution of a Setup on a Task, producing reproducible results and evaluations.

OpenML’s core innovation is its API-first approach. By integrating directly into popular toolboxes like Scikit-learn, Weka, and mlR, it allows researchers to upload results and associated metadata automatically from their native environments, lowering the friction of sharing to nearly zero.

Dataset Characterization and Metafeatures

To facilitate metalearning, OpenML serves as a meta-repository hosting thousands of datasets. To characterize these datasets, the platform automatically computes a wide array of metafeatures. These range from simple counts to complex "landmarkers" that use the performance of simple models to probe a dataset's inherent difficulty.

Table 1: Standard metafeatures available in OpenML, categorized by type.

By defining standardized "Tasks," OpenML ensures that results from different researchers are directly comparable, as they utilize the exact same data splits and evaluation metrics.

Key Insights and Results

OpenML transforms the "exhaust" of daily research into a structured dataset for training future AI. By treating an entire experiment (dataset + algorithm + result) as a single data point, the platform creates a "world machine learning information" map.

One of the most significant benefits is automated evaluation. OpenML calculates detailed metrics (precision, recall, ROC curves) on the server side based on uploaded predictions. This ensures that metrics are calculated consistently across all experiments, regardless of the library used to generate the model.

Figure 1: Example of an OpenML run (Run 24996) identifying the specific experiment instance.

Limitations and Future Directions

While OpenML provides a robust framework for supervised learning, several challenges remain. Dependency management is a persistent hurdle; ensuring total reproducibility is difficult if underlying system libraries or hardware architectures change. Additionally, while supervised classification and regression are well-supported, areas like Reinforcement Learning still lack the rich, organized metadata structures found in OpenML. Finally, the utility of metalearning depends on the quality of the metadata; the community must adopt rigorous practices to ensure the data pushed to these platforms is accurate and finely detailed.

Summary updated

Abstract

Summary This chapter presents a review of online repositories where researchers can share data, code, and experiments. In particular, it covers OpenML, an online platform for sharing and organizing machine learning data automatically. OpenML contains thousands of datasets and algorithms, and millions of experimental results. We describe the basic philosophy involved, and its basic components: datasets, tasks, flows, setups, runs, and benchmark suites. OpenML has API bindings in various programming languages, making it easy for users to interact with the API in their native language. One important feature of OpenML is the integration into various machine learning toolboxes, such as Scikit-learn, Weka, and mlR. Users of these toolboxes can automatically upload all their results, leading to a large repository of experimental results.
BibTeX
@inproceedings{brazdil2022metadata,
  author = {Pavel Brazdil and Jan N. van Rijn and Carlos Soares and Joaquin Vanschoren},
  title = {{Metadata Repositories}},
  booktitle = {Cognitive},
  year = {2022},
  doi = {10.1007/978-3-030-67024-5\_16},
}
Figure