MtL-NFW: A Meta-Learning Framework for Automated Noise Filter Selection and Hyperparameter Optimization in Auto-ML

Irfan Ali Khan, Xianchao Zhang, Ramesh Kumar, Saadat M. Alhashmi, Rahman AliResearch Square 2024 · DOI

TL;DR

MtL-NFW is a new meta-learning framework designed to automate the selection of noise filters and their hyperparameters for datasets with "noisy" labels. By training a meta-learner on realistic, boundary-based noise patterns rather than random flips, the system achieves a 69% success rate in recommending near-optimal filtering workflows, significantly outperforming existing Auto-ML benchmarks.

What Problem Does This Paper Address?

In real-world machine learning, data is rarely perfect. "Class label noise"—where instances are assigned the wrong category—can severely degrade model performance. While many noise-filtering algorithms exist to "clean" this data, choosing the right filter for a specific dataset is a difficult "Algorithm Selection Problem" (ASP).

Current Automated Machine Learning (Auto-ML) solutions for noise filtering have two major flaws:

  1. Ignoring Hyperparameters: Most systems pick a filter but ignore its internal settings (hyperparameters), which can be the difference between a filter being too strict (deleting good data) or too lenient (leaving noise in).
  2. Unrealistic Training Data: To train a meta-learner, researchers usually inject "random" noise into clean datasets. However, real-world noise isn't random; it usually occurs near the "decision boundaries" where classes overlap. Meta-learners trained on random noise often fail when faced with complex, real-world distributions.

Methods

The authors propose MtL-NFW (Meta-learning for Noise-Filter Workflow Selection), which treats the task as a CASH (Combined Algorithm Selection and Hyperparameter Optimization) problem.

Key technical components include:

  • Neighbor-wise Noise Imputation: Instead of random flipping, the authors used a "noise susceptibility score" based on Euclidean distances to nearest neighbors. This targets instances near class boundaries, creating a more robust and realistic meta-training set.
  • Workflow Construction: They evaluated 453 distinct workflows per dataset. Each workflow consists of a base classifier (Random Forest) paired with one of six noise filters (EBF, EWF, ORBF, HARF, DCF, AENN) and specific hyperparameter configurations.
  • Meta-Feature Extraction: The system extracts 6 groups of meta-features (Statistical, Distance-Based, Information-Theoretic, Topological, Complexity, and Noise-Specific) to characterize each dataset.
  • The Meta-Learner: An XGBoost model was trained to map these meta-features to workflow performance (measured by F1-score). When given a new dataset, it predicts and ranks the top 20 most effective workflows.

Datasets

The study utilized 114 classification datasets sourced from the OpenML repository. By running 453 workflows across these datasets, the authors generated a massive meta-knowledge base of 51,642 unique configurations to train and validate their meta-learner.

Key Insights and Ideas

  • The "Oracle" Benchmark: The authors identified the "Oracle" solution (the absolute best possible filter/hyperparameter combo) for every dataset. They found that no single filter consistently wins, proving that automated, context-aware selection is mandatory.
  • Boundary Noise is Better: By using the Neighbor-wise method, the meta-learner "learns" to recognize the types of noise that actually confuse classifiers in the real world, leading to more accurate recommendations.
  • The Top-20 Efficiency: The researchers found that recommending the top 20 workflows provides an optimal "elbow" point—it maximizes the probability of finding a near-optimal solution while drastically reducing the computational cost compared to an exhaustive search.

Key Results

The framework was evaluated using a leave-one-out cross-validation strategy and compared against four state-of-the-art (SOTA) methods:

  • Win Probability: MtL-NFW achieved a 0.69 win probability, meaning that in 69% of cases, its recommendations were "practically equivalent" to the Oracle solution (within a ±1% F1-score margin).
  • Competitive Edge: It significantly outperformed all baselines, including M-LNA (0.54), M-RSS (0.51), M-NFER (0.41), and M-Ranker (0.38).
  • Statistical Significance: Z-tests confirmed that MtL-NFW’s performance gains were statistically significant (p < 0.05) across all comparisons.

Limitations and Future Work

While MtL-NFW is a significant step forward, the authors note a few areas for expansion:

  • Base Classifier Variety: The current study relies solely on Random Forest as the base classifier. Future versions could include Support Vector Machines (SVM), Gradient Boosting, or Neural Networks.
  • Noise Types: The framework focuses on class label noise. The authors suggest exploring "attribute-level noise" (errors in the features themselves) in future iterations.
  • Active Learning: Integrating active learning could help the framework adapt even more quickly to entirely new patterns of noise or highly imbalanced class distributions.

Summary updated

Abstract

<title>Abstract</title> The extensive implementation of machine learning (ML) has transformed data analysis and decision-making processes. However, the process of choosing suitable ML algorithms for a specific task remains challenging, commonly referred to as the Algorithm Selection Problem (ASP) in academic literature. Automated machine learning (Auto-ML) tackles algorithm selection and hyperparameter adjustments by automating these processes, hence decreasing the required time and expertise. The current study is focused on the automated selection of noise filters for the task of classifying data with noisy labels. Class label noise, resulting from erroneous data labels, has a greater effect on the learning process compared to attribute-level noise. The existing Auto-ML approaches for selecting suitable noise filters overlook hyperparameter adjustment, which is important for the effectiveness of noise filters. Instead, it relies on computationally demanding techniques such as grid search. In addition, the current Auto-ML algorithms utilize random noise imputation techniques in their methodologies that are not 1 able to adequately capture the complex noise distributions typically present in real-world datasets. In order to overcome these limitations, we propose the introduction of Meta-learning for Noise-Filter Workflow Selection (MtL-NFW), an innovative framework that automate and integrate adequate hyperparameter selection along with noise filter for a certain task. This eliminates the need to rely on manual and computationally intensive methods for hyperparameters settings. Our second contribution is to leverage and employ the Neighbor-wise noise imputation technique to our framework, that induce noise based on proximity to class boundaries, generating more robust meta-training data. Our framework evaluates 114 classification datasets with six noise filtering algorithms and Random Forest as the base classifier, extracting six groups of meta-features and generating a meta-knowledge base of 51,642 configurations. XGBoost trained as meta-learner, predicts and recommends optimal noise filter and hyperparameter combinations for new datasets. Empirical validation on 114 OpenML datasets shows that MtL-NFW significantly improves the accuracy and robustness of noise filter recommendations, outperforming four state-of-the-art Auto-ML methods.
BibTeX
@inproceedings{khan2024mtl,
  author = {Irfan Ali Khan and Xianchao Zhang and Ramesh Kumar and Saadat M. Alhashmi and Rahman Ali},
  title = {{MtL-NFW: A Meta-Learning Framework for Automated Noise Filter Selection and Hyperparameter Optimization in Auto-ML}},
  booktitle = {Research Square},
  year = {2024},
  doi = {10.21203/rs.3.rs-4638344/v1},
}
Figure