In machine learning, we often transform categorical data—like "City" or "Color"—into a series of binary "one-hot" indicators. While this makes the data digestible for algorithms like XGBoost or Random Forests, it creates a fundamental crisis for explainability.
When we ask an XAI (Explainable AI) tool to tell us which features were important, it typically treats every binary bit as an independent actor. This fragmentation leads to two major problems: it dilutes the perceived importance of categorical variables and creates "impossible" states that break the mathematical logic of the explanation.
The Fragmentation Problem
Imagine a model predicting house prices using a "Neighborhood" feature. If this feature is one-hot encoded into 50 different binary columns, a standard attribution method like SHAP will assign a small amount of credit to each individual neighborhood.
C. Bayan Bruss, Brian Barr, and Samuel Sharpe argue in BASED-XAI: Breaking Ablation Studies Down for Explainable Artificial Intelligence that this fragmentation hides the true drivers of a model. When evaluating feature importance through ablation (removing features to see how the model reacts), treating bits independently results in illogical curves. The model isn't reacting to "Neighborhood"; it’s reacting to a single, tiny binary indicator, which makes the categorical variable appear far less significant than it actually is.
The Mathematical Fallacy of Summing Bits
A common "fix" is to simply sum the importance scores of all binary indicators belonging to the same category. However, Salim I. Amoukou, Nicolas J.-B. Brunel, and Tangi Salaün demonstrate in Accurate and robust Shapley Values for explaining predictions and focusing on local important variables that this is theoretically flawed.
Shapley values are based on cooperative game theory, where "players" (features) form coalitions. If you split one feature into multiple binary indicators, you change the structure of the game itself. Summing the results of the "split" game does not equal the result of the "original" game. This inconsistency can lead to sign reversals—where a category that actually increases a prediction appears to decrease it because its importance was split across dependent indicators.
Grouping as a Logical Necessity
The solution emerging across this research is to treat related binary indicators as a single logical unit—a coalition—during the attribution process itself.
In Understanding Interventional TreeSHAP : How and Why it Works, Gabriel Laberge and Yann Pequignot provide the formal proof for why this grouping is necessary. By treating a set of one-hot variables as a single entity, we ensure that the "interventional" logic of SHAP remains sound.
When the algorithm "toggles" a feature to see its effect, it must toggle the entire categorical set. If "Color" changes from Red to Blue, the "Red" bit must flip to 0 and the "Blue" bit must flip to 1 simultaneously. If we toggle them independently, we force the model to evaluate a state where a house is both Red and Blue (or neither), which is an "impossible" state that the model never saw during training.
Why It Works: Maintaining Manifold Integrity
The core intuition is that features in a model do not exist in a vacuum; they exist on a "manifold" of valid data. One-hot encoding creates strict dependencies: if one bit is 1, the others must be 0.
- Preventing Confusion: By grouping features, we prevent the model from being evaluated on "impossible" data points, which Bruss et al. identify as a primary source of error in XAI validation.
- Restoring Game Integrity: By treating the group as a single player in the Shapley game, Amoukou et al. show we can achieve "encoding invariance"—meaning the importance of "City" remains the same whether it was one-hot encoded, dummy encoded, or left as a raw string.
- Accurate Ranking: Grouping ensures that high-cardinality categorical variables (those with many levels) receive their fair share of credit, allowing them to correctly appear at the top of feature importance rankings.
By moving from "bit-level" attribution to "entity-level" attribution, these researchers have established a more theoretically sound foundation for explaining the complex tabular models used throughout the industry.