EIX: Explain Interactions in 'XGBoost'

Szymon Maksymiuk, Ewelina Karbowiak, Przemysław Biecek, Przemyslaw BiecekOpenalex Unknown 2019 · DOI · OpenAlex

Abstract

Structure mining from 'XGBoost' and 'LightGBM' models. Key functionalities of this package cover: visualisation of tree-based ensembles models, identification of interactions, measuring of variable importance, measuring of interaction importance, explanation of single prediction with break down plots (based on 'xgboostExplainer' and 'iBreakDown' packages). To download 'LightGBM' use the following link: https://github.com/Microsoft/LightGBM. 'EIX' is a part of the 'DrWhy.AI' universe.

Introduction

Gradient Boosting Machines (GBMs) like XGBoost and LightGBM are the workhorses of modern tabular data science. They are incredibly powerful but notoriously difficult to interpret. While practitioners often look at "Feature Importance" scores, these numbers rarely tell the whole story—especially when features work together in complex ways. The R package EIX (Explain Interactions in 'XGBoost') aims to bridge this gap by providing a specialized toolkit for mining the internal structures of these models.

The Interpretability Gap in Tree Ensembles

In a standard decision tree, an interaction is easy to see: if the model splits on "Age" and then immediately on "Income," those two variables are interacting. However, in an ensemble of hundreds of trees, these interactions become buried. Most practitioners rely on global importance measures like "Gain" or "Frequency," but these metrics often collapse the complexity of the model into a single list of variables. They fail to answer critical questions about which variables consistently work together or which dominate the "roots" of the trees versus the leaves.

Methodology: Structure Mining

EIX employs several novel techniques for "structure mining" within tree-based ensembles:

  • Interaction Detection: The package identifies interactions by analyzing parent-child relationships within the trees. If a child node splits on a different variable than its parent, EIX flags this as a potential interaction.
  • Multi-Metric Importance: Instead of a single score, EIX calculates a variety of measures, including sumGain (total contribution), sumCover (number of observations affected), meanDepth (proximity to the root), and frequency.
  • Waterfall/Break-down Plots: Building on the iBreakDown and xgboostExplainer frameworks, EIX generates local explanations that attribute a specific prediction's value to individual features and their interactions.

Datasets and Case Studies

The package's utility is demonstrated using classic datasets such as the Titanic survival data and HR analytics data.

The HR_data dataset is used to explore why high-performing employees leave organizations prematurely.

The HR dataset (14,999 rows) allows for the prediction of employee attrition based on factors like satisfaction level and salary, providing a rich ground for testing interaction detection and model transparency.

Key Insights: The Radar View and Root Dominance

EIX treats feature interactions as first-class citizens, assigning them their own importance scores. One of the most powerful features is the "Radar View," which allows researchers to see the multi-dimensional profile of a variable.

Configuration parameters for the radar plot, which visualizes six measures of variable or interaction importance.

By using radar plots, EIX enables users to identify "silver bullet" predictors—variables that might have low frequency but extremely high gain. Furthermore, the package emphasizes "root dominance," identifying which variables capture the most significant variance across the entire ensemble by appearing frequently at the initial splits of the trees.

Conclusion

EIX provides a unified API that works seamlessly across both XGBoost and LightGBM. While it is currently tailored specifically for these two frameworks, it offers a necessary and sophisticated lens into the heart of the ensemble for researchers and engineers looking to move beyond simple accuracy metrics.

Summary updated

Abstract

Structure mining from 'XGBoost' and 'LightGBM' models. Key functionalities of this package cover: visualisation of tree-based ensembles models, identification of interactions, measuring of variable importance, measuring of interaction importance, explanation of single prediction with break down plots (based on 'xgboostExplainer' and 'iBreakDown' packages). To download the 'LightGBM' use the following link: &lt;<a href="https://github.com/Microsoft/LightGBM" target="_top">https://github.com/Microsoft/LightGBM</a>&gt;. 'EIX' is a part of the 'DrWhy.AI' universe.
BibTeX
@inproceedings{maksymiuk2019eix,
  author = {Szymon Maksymiuk and Ewelina Karbowiak and Przemysław Biecek and Przemyslaw Biecek},
  title = {{EIX: Explain Interactions in 'XGBoost'}},
  booktitle = {Openalex Unknown},
  year = {2019},
  doi = {10.32614/cran.package.eix},
}
Figure