Abstract
Structure mining from 'XGBoost' and 'LightGBM' models. Key functionalities of this package cover: visualisation of tree-based ensembles models, identification of interactions, measuring of variable importance, measuring of interaction importance, explanation of single prediction with break down plots (based on 'xgboostExplainer' and 'iBreakDown' packages). To download 'LightGBM' use the following link: https://github.com/Microsoft/LightGBM. 'EIX' is a part of the 'DrWhy.AI' universe.
Introduction
Gradient Boosting Machines (GBMs) like XGBoost and LightGBM are the workhorses of modern tabular data science. They are incredibly powerful but notoriously difficult to interpret. While practitioners often look at "Feature Importance" scores, these numbers rarely tell the whole story—especially when features work together in complex ways. The R package EIX (Explain Interactions in 'XGBoost') aims to bridge this gap by providing a specialized toolkit for mining the internal structures of these models.
The Interpretability Gap in Tree Ensembles
In a standard decision tree, an interaction is easy to see: if the model splits on "Age" and then immediately on "Income," those two variables are interacting. However, in an ensemble of hundreds of trees, these interactions become buried. Most practitioners rely on global importance measures like "Gain" or "Frequency," but these metrics often collapse the complexity of the model into a single list of variables. They fail to answer critical questions about which variables consistently work together or which dominate the "roots" of the trees versus the leaves.
Methodology: Structure Mining
EIX employs several novel techniques for "structure mining" within tree-based ensembles:
- Interaction Detection: The package identifies interactions by analyzing parent-child relationships within the trees. If a child node splits on a different variable than its parent, EIX flags this as a potential interaction.
- Multi-Metric Importance: Instead of a single score, EIX calculates a variety of measures, including
sumGain(total contribution),sumCover(number of observations affected),meanDepth(proximity to the root), andfrequency. - Waterfall/Break-down Plots: Building on the
iBreakDownandxgboostExplainerframeworks, EIX generates local explanations that attribute a specific prediction's value to individual features and their interactions.
Datasets and Case Studies
The package's utility is demonstrated using classic datasets such as the Titanic survival data and HR analytics data.
The HR dataset (14,999 rows) allows for the prediction of employee attrition based on factors like satisfaction level and salary, providing a rich ground for testing interaction detection and model transparency.
Key Insights: The Radar View and Root Dominance
EIX treats feature interactions as first-class citizens, assigning them their own importance scores. One of the most powerful features is the "Radar View," which allows researchers to see the multi-dimensional profile of a variable.
By using radar plots, EIX enables users to identify "silver bullet" predictors—variables that might have low frequency but extremely high gain. Furthermore, the package emphasizes "root dominance," identifying which variables capture the most significant variance across the entire ensemble by appearing frequently at the initial splits of the trees.
Conclusion
EIX provides a unified API that works seamlessly across both XGBoost and LightGBM. While it is currently tailored specifically for these two frameworks, it offers a necessary and sophisticated lens into the heart of the ensemble for researchers and engineers looking to move beyond simple accuracy metrics.
Summary updated