FIND: Explainable Framework for Meta-learning

Xinyue Shao, Hongzhi Wang, Xiao Zhu, Feng XiongarXiv 2022 · DOI · OpenReview

Introduction

Meta-learning, often described as "learning to learn," has become a cornerstone of automated machine learning (AutoML). Its primary application is automated algorithm selection: predicting which machine learning model—such as a Random Forest or Support Vector Machine—will yield the best performance for a given dataset. However, these meta-learners are frequently "black boxes," leading to two significant challenges. First, there is a lack of meta-explainability, where experts cannot discern why a specific algorithm was recommended. Second, there is a lack of recommendation explainability, where the chosen algorithm's internal logic remains opaque, especially when dealing with complex real-world feature dependencies.

The FIND framework (Feature Influence iNterpretation framework for meta-learning Decision models) addresses these hurdles by integrating deep learning with causal reasoning. FIND provides transparency by explaining algorithm selection and generating realistic "what-if" (counterfactual) scenarios that respect real-world constraints.

The FIND Framework

The FIND framework is structured into three distinct modules designed to provide end-to-end interpretability in the meta-learning pipeline.

The FIND framework architecture, illustrating the three core modules: Model Recommendation, Feature Influence Computation, and Counterfactual Generation.

1. Model Recommendation & Meta-Explainability

The system employs a neural network to map a dataset’s "meta-features"—such as entropy, number of attributes, or geometric mean—to the optimal algorithm. To demystify this choice, FIND utilizes Integrated Gradients. This technique attributes the network's output to specific input meta-features, revealing the dataset characteristics that drive the recommendation.

2. Feature Influence Computation

Unlike standard methods that perturb features in isolation, FIND incorporates Structural Causal Models (SCM). When a feature is adjusted, the system uses a causal graph to update all related latent factors. This ensures that the resulting influence scores reflect the feature's behavior within a complex, interconnected system rather than an artificial vacuum.

3. Causal Counterfactual Generation

To explain specific model decisions (e.g., a loan denial), FIND generates counterfactuals—the smallest changes required to alter an outcome. By using a greedy search algorithm that adheres to causal constraints, FIND ensures that suggested changes (like increasing income) are realistic and logically consistent with other variables (like occupation).

Experimental Evaluation

The researchers evaluated FIND using a custom-built Metadata Set containing 23 meta-features from various OpenML tasks to train the recommendation engine. For testing feature influence and counterfactuals, they utilized the Adult (Census Income) and German-Credit datasets, which are standard benchmarks for fairness and interpretability.

Key Insights and Results

The integration of causality serves as a "truth filter," preventing the generation of mathematically possible but logically impossible explanations. FIND demonstrated that accuracy and explainability are not mutually exclusive; the framework achieved a recommendation accuracy of 86.60%, significantly outperforming baselines like Random Forest (61.61%) and XGBoost (75.00%).

Integrated Gradients for the Explanation-recommend Module, showing how different meta-features influence the selection of specific machine learning algorithms.

As shown in the analysis of Integrated Gradients, specific patterns emerge: AdaBoostM1 is frequently recommended for multi-class tasks with high target category counts, while Random Forests show a strong affinity for datasets with numerous categorical features.

Qualitative tests on the German-Credit dataset revealed that FIND’s feature influence scores captured domain-specific logic—such as the interplay between age, housing, and credit risk—more accurately than traditional tools like LIME or SHAP. Furthermore, while FIND’s counterfactuals were slightly more "distant" from the original data points than those from other methods, they were rated as significantly more rational because they accounted for downstream causal effects.

Limitations and Future Directions

Despite its success, FIND relies on pre-defined causal graphs (SCMs), which can be difficult to construct in some business contexts. Future work aims to expand the framework into more complex domains, such as time-series meta-learning, and explore automated causal discovery directly from data to reduce dependency on expert-defined models.

Summary updated

Abstract

Meta-learning is used to efficiently enable the automatic selection of machine learning models by combining data and prior knowledge. Since the traditional meta-learning technique lacks explainability, as well as shortcomings in terms of transparency and fairness, achieving explainability for meta-learning is crucial. This paper proposes FIND, an interpretable meta-learning framework that not only can explain the recommendation results of meta-learning algorithm selection, but also provide a more complete and accurate explanation of the recommendation algorithm's performance on specific datasets combined with business scenarios. The validity and correctness of this framework have been demonstrated by extensive experiments.
BibTeX
@misc{shao2022find,
  author = {Xinyue Shao and Hongzhi Wang and Xiao Zhu and Feng Xiong},
  title = {{FIND: Explainable Framework for Meta-learning}},
  year = {2022},
  eprint = {2205.10362},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.10362},
}
Figure