Advances in Metalearning: ECML/PKDD Workshop on Meta-Knowledge Transfer

Pavel Brazdil, Jan N. van Rijn, Henry Gouk, Felix MohrMeta-Knowledge Transfer @ ECML/PKDD 2022 · OpenReview

Abstract

Meta-knowledge plays a critical role in contemporary machine learning and Automated Machine Learning (AutoML) systems. By observing learning processes—either on the same task or across different tasks—systems can represent and store meta-knowledge to optimize future learning. Metalearning systems typically explore this knowledge to facilitate "warm starts" or dynamic scheduling within search spaces. This paper summarizes the contributions of the 2022 ECML/PKDD Workshop on Meta-Knowledge Transfer, which addressed how meta-knowledge is acquired, represented, and exploited. Key highlights include invited talks by Timothy Hospedales on knowledge transfer and Pascal Hitzler on neuro-symbolic AI, alongside various methodological advancements in hyperparameter optimization and neural architecture search.

Introduction

Standard machine learning paradigms often treat every new task as an isolated problem, starting from a "blank slate." Metalearning, or "learning to learn," seeks to overcome this inefficiency by leveraging meta-knowledge—insights extracted from previous learning experiences regarding which algorithms, architectures, or hyperparameters perform best on specific data distributions.

Despite significant progress, the field faces several persistent challenges:

  1. Domain Shift: Performance degradation when training and testing tasks originate from different distributions.
  2. Computational Efficiency: The high cost of Hyperparameter Optimization (HPO) and Neural Architecture Search (NAS).
  3. Explainability: The "black box" nature of deep metalearning models.
  4. Interoperability: The lack of standardized protocols to share meta-knowledge between different AutoML platforms.

Methodological Innovations

The workshop showcased several novel approaches designed to address these bottlenecks across four primary themes:

Researchers introduced BOingG, a two-stage procedure for Bayesian Optimization. This method utilizes a Random Forest to map the global structure of a search space before employing Gaussian processes to refine local, promising regions. In the realm of Neural Architecture Search (NAS), efficiency was improved through Embedding Proximity. Rather than training a performance predictor from scratch, this approach uses k-Nearest Neighbor (kNN) logic within an embedded architecture space, significantly accelerating the estimation of new model performance.

Interpretability and Neuro-Symbolic Integration

To address the need for transparency, researchers presented Interpretable Meta-Scoring, which uses logistic regression to model the probability of one algorithm outperforming another. This allows for the direct interpretation of meta-score coefficients. Furthermore, the integration of Neuro-Symbolic AI—combining symbolic logic (ontologies) with deep learning—was proposed to enable formal reasoning and human-correctable meta-knowledge.

Instance Selection and Combinatorial Problems

For complex problems such as Satisfiability (SAT) and Mixed Integer Programming (MIP), methods were adapted to identify the most "informative" data instances. This instance selection helps in configuring solvers more efficiently by focusing on data that best represents the problem's complexity.

Benchmarks and Data Diversity

The advancement of metalearning relies on diverse, high-quality benchmarks. A significant contribution highlighted was Meta-Album, a meta-dataset featuring 40 image classification datasets across 10 domains (e.g., manufacturing, botany, and medicine). Other benchmarks utilized by workshop contributors included OpenML and VTAB for algorithm selection, and CIFAR-10 for NAS efficiency testing.

Key Results and Insights

  • Efficiency Gains: Instance selection methods for SAT and MIP problems demonstrated remarkable speedups, reaching decisions 5 to 3,000 times faster than random sampling.
  • Predictive Accuracy: The kNN-based embedding method for NAS achieved state-of-the-art performance prediction while maintaining a lower computational footprint than traditional neural predictors.
  • Workflow Simplification: In classification pipelines, identifying and removing infrequent outlier-handling methods reduced workflow portfolio complexity by over 50% without a significant loss in accuracy.
  • The "Silo" Problem: A core insight from the workshop is that meta-knowledge remains largely trapped within individual systems. While HPO landscapes are increasingly navigable, the field must move toward automating the definition of configuration spaces, which are currently defined by human experts.

Conclusion and Future Directions

The workshop identified a critical "missing link" in the metalearning ecosystem: the ability to transfer meta-knowledge between disparate AutoML systems. Future research must focus on developing protocols that allow AutoML systems to act as "agents" capable of communicating their experiences.

Moving forward, the integration of dynamic scheduling—where systems decide in real-time which areas of a search space to prioritize—and the continued fusion of neuro-symbolic AI will be essential. These steps will bridge the gap between raw computational performance and the need for human-understandable, interoperable AI reasoning.

Summary updated

Abstract

Meta-knowledge plays an important role in current machine learning and AutoML systems. One way of acquiring meta-knowledge is by observing learning processes (on the same task, or on different tasks) and representing it in such a way that it can be used later to improve future learning processes. Metalearning systems, on the other hand, normally explore metaknowledge acquired on different problems. The systems may, in addition, use metaknowledge concerning which part of the space should be examined first (i.e., a warm start or dynamic scheduling). Various contributions of this workshop addressed various aspects of metaknowledge, and in particular, how it is exploited in different systems. This workshop included two invited talks, one by Hospedales on “Meta-learning for Knowledge Transfer” and another by Hitzler on “Some advances regarding ontologies and neuro-symbolic artificial intelligence”.
BibTeX
@inproceedings{brazdil2022advances,
  author = {Pavel Brazdil and Jan N. van Rijn and Henry Gouk and Felix Mohr},
  title = {{Advances in Metalearning: ECML/PKDD Workshop on Meta-Knowledge Transfer}},
  booktitle = {Meta-Knowledge Transfer @ ECML/PKDD},
  year = {2022},
  url = {https://openreview.net/forum?id=OdsIVmXfg5},
}
Figure