Abstract
Meta-knowledge plays a critical role in contemporary machine learning and Automated Machine Learning (AutoML) systems. By observing learning processes—either on the same task or across different tasks—systems can represent and store meta-knowledge to optimize future learning. Metalearning systems typically explore this knowledge to facilitate "warm starts" or dynamic scheduling within search spaces. This paper summarizes the contributions of the 2022 ECML/PKDD Workshop on Meta-Knowledge Transfer, which addressed how meta-knowledge is acquired, represented, and exploited. Key highlights include invited talks by Timothy Hospedales on knowledge transfer and Pascal Hitzler on neuro-symbolic AI, alongside various methodological advancements in hyperparameter optimization and neural architecture search.
Introduction
Standard machine learning paradigms often treat every new task as an isolated problem, starting from a "blank slate." Metalearning, or "learning to learn," seeks to overcome this inefficiency by leveraging meta-knowledge—insights extracted from previous learning experiences regarding which algorithms, architectures, or hyperparameters perform best on specific data distributions.
Despite significant progress, the field faces several persistent challenges:
- Domain Shift: Performance degradation when training and testing tasks originate from different distributions.
- Computational Efficiency: The high cost of Hyperparameter Optimization (HPO) and Neural Architecture Search (NAS).
- Explainability: The "black box" nature of deep metalearning models.
- Interoperability: The lack of standardized protocols to share meta-knowledge between different AutoML platforms.
Methodological Innovations
The workshop showcased several novel approaches designed to address these bottlenecks across four primary themes:
Hierarchical Optimization and Search
Researchers introduced BOingG, a two-stage procedure for Bayesian Optimization. This method utilizes a Random Forest to map the global structure of a search space before employing Gaussian processes to refine local, promising regions. In the realm of Neural Architecture Search (NAS), efficiency was improved through Embedding Proximity. Rather than training a performance predictor from scratch, this approach uses k-Nearest Neighbor (kNN) logic within an embedded architecture space, significantly accelerating the estimation of new model performance.
Interpretability and Neuro-Symbolic Integration
To address the need for transparency, researchers presented Interpretable Meta-Scoring, which uses logistic regression to model the probability of one algorithm outperforming another. This allows for the direct interpretation of meta-score coefficients. Furthermore, the integration of Neuro-Symbolic AI—combining symbolic logic (ontologies) with deep learning—was proposed to enable formal reasoning and human-correctable meta-knowledge.
Instance Selection and Combinatorial Problems
For complex problems such as Satisfiability (SAT) and Mixed Integer Programming (MIP), methods were adapted to identify the most "informative" data instances. This instance selection helps in configuring solvers more efficiently by focusing on data that best represents the problem's complexity.
Benchmarks and Data Diversity
The advancement of metalearning relies on diverse, high-quality benchmarks. A significant contribution highlighted was Meta-Album, a meta-dataset featuring 40 image classification datasets across 10 domains (e.g., manufacturing, botany, and medicine). Other benchmarks utilized by workshop contributors included OpenML and VTAB for algorithm selection, and CIFAR-10 for NAS efficiency testing.
Key Results and Insights
- Efficiency Gains: Instance selection methods for SAT and MIP problems demonstrated remarkable speedups, reaching decisions 5 to 3,000 times faster than random sampling.
- Predictive Accuracy: The kNN-based embedding method for NAS achieved state-of-the-art performance prediction while maintaining a lower computational footprint than traditional neural predictors.
- Workflow Simplification: In classification pipelines, identifying and removing infrequent outlier-handling methods reduced workflow portfolio complexity by over 50% without a significant loss in accuracy.
- The "Silo" Problem: A core insight from the workshop is that meta-knowledge remains largely trapped within individual systems. While HPO landscapes are increasingly navigable, the field must move toward automating the definition of configuration spaces, which are currently defined by human experts.
Conclusion and Future Directions
The workshop identified a critical "missing link" in the metalearning ecosystem: the ability to transfer meta-knowledge between disparate AutoML systems. Future research must focus on developing protocols that allow AutoML systems to act as "agents" capable of communicating their experiences.
Moving forward, the integration of dynamic scheduling—where systems decide in real-time which areas of a search space to prioritize—and the continued fusion of neuro-symbolic AI will be essential. These steps will bridge the gap between raw computational performance and the need for human-understandable, interoperable AI reasoning.
Summary updated