BoostEMM - Transparent Boosting using Exceptional Model Mining

Simon van der Zon, Oren Zeev-Ben-Mordehai, Tom Vrijdag, Werner van Ipenburg, Wouter Duivesteijn, Jan Veldsink, Mykola Pechenizkiy

Abstract

Boosting algorithms like AdaBoost are highly effective but often suffer from overfitting to noise and a lack of interpretability. We present BoostEMM, a novel variant of AdaBoost that integrates Exceptional Model Mining (EMM) to prioritize learning from coherent subgroups of data rather than individual misclassified records. By focusing on interpretable subgroups, BoostEMM provides human-readable explanations for the boosting process while maintaining competitive performance and robustness against outliers.

Introduction

Ensemble learning methods, particularly boosting, have become staples in machine learning due to their high predictive power. AdaBoost, for instance, iteratively trains weak classifiers by increasing the weights of misclassified records. However, this record-level focus leads to two significant drawbacks: overfitting to noise and the "black box" nature of the resulting model. In high-stakes domains like finance or healthcare, the inability to explain why a model prioritizes certain data points is a major barrier to adoption.

BoostEMM addresses these challenges by ensuring the model focuses on groups of records that share common, interpretable characteristics. This approach effectively "kills two birds with one stone": it filters out isolated noise and provides a rationale for the model's focus during training.

Methodology: Subgroup-Based Boosting

The core innovation of BoostEMM is the replacement of individual record weighting with a subgroup-based approach. Instead of boosting every misclassified record, the algorithm employs EMM to identify the top-$q$ subgroups where the current model performs poorly.

Subgroup Selection and Quality Measures

BoostEMM uses a beam search to find "exceptional" subgroups defined by conjunctions of features. The definition of "exceptional" is steerable through four distinct quality measures ($\phi$):

  • $\phi_{err}$: Focuses on high error rates, mimicking standard AdaBoost.
  • $\phi_{\kappa}$: Utilizes the Kappa statistic to address class imbalance.
  • $\phi_{FNR}$ / $\phi_{FPR}$: Targets False Negatives or False Positives, making it ideal for cost-sensitive tasks like fraud detection.

Transparency and Visualization

A key feature of BoostEMM is its ability to visualize the boosting process through a "tree of literals." Each node in this tree represents a descriptive constraint (e.g., age < 30), with the node size indicating the constraint's contribution to the boosting process.

Figure 1: The 'tree of literals' visualization in BoostEMM, illustrating how descriptive constraints like age, capital-gain, and marital status define the subgroups targeted during boosting.

Experimental Evaluation

The researchers evaluated BoostEMM across three binary classification tasks: the UCI Adult dataset, a Kaggle Credit-card Fraud dataset, and a proprietary Rabobank Fraud dataset containing over 30 million transactions.

Performance and Interpretability

BoostEMM demonstrated competitive accuracy compared to standard AdaBoost. Notably, in fraud detection tasks, BoostEMM using the False Negative measure ($\phi_{FNR}$) achieved an AUC of 0.907, significantly outperforming standard AdaBoost's 0.852.

Beyond performance, the model successfully extracted meaningful rules. For example, it identified subgroups such as capital-gain ≥ 6896.48 ∧ age ≥ 19.52 as high-priority areas for learning. On the proprietary Rabobank data, domain experts confirmed that the discovered subgroups aligned with known fraud "modus operandi," validating the algorithm's ability to capture real-world behaviors.

Discussion and Future Work

By forcing the model to boost groups defined by rules, BoostEMM naturally ignores isolated noisy outliers. This "accountability by design" allows stakeholders to understand the reasoning behind model decisions.

However, the algorithm faces a "cold start" problem with certain quality measures. If an initial weak learner is extremely poor, cost-sensitive measures like $\phi_{FPR}$ may fail to find evaluable subgroups. Future research aims to "dovetail" different boosting types—such as starting with error-based boosting and transitioning to cost-sensitive measures—to leverage the strengths of multiple quality measures within a single training run.

Conclusion

BoostEMM represents a significant step toward making ensemble learning both transparent and robust. By shifting the focus from individual records to interpretable subgroups, it provides a framework that is not only powerful but also accountable to human experts.

Summary updated

BibTeX
@inproceedings{zon2017boostemm,
  author = {Simon van der Zon and Oren Zeev-Ben-Mordehai and Tom Vrijdag and Werner van Ipenburg and Wouter Duivesteijn and Jan Veldsink and Mykola Pechenizkiy},
  title = {{BoostEMM - Transparent Boosting using Exceptional Model Mining}},
  booktitle = {MIDAS@PKDD/ECML},
  year = {2017},
}
Figure