Abstract
Interpretability is a pressing issue for decision systems. Many post hoc methods have been proposed to explain the predictions of a single machine learning model. However, business processes and decision systems are rarely centered around a unique model. These systems combine multiple models that produce key predictions, and then apply decision rules to generate the final decision. To explain such decisions, we propose the Semi-Model-Agnostic Contextual Explainer (SMACE), a new interpretability method that combines a geometric approach for decision rules with existing interpretability methods for machine learning models to generate an intuitive feature ranking tailored to the end user. We show that established model-agnostic approaches produce poor results on tabular data in this setting, in particular giving the same importance to several features, whereas SMACE can rank them in a meaningful way.
Introduction
In the field of Explainable AI (XAI), standard tools like SHAP and LIME are designed to interpret individual machine learning (ML) models. However, real-world decision-making—particularly in regulated industries like finance and healthcare—often relies on composite systems. These systems integrate one or more ML models with explicit, human-defined business rules. For instance, a loan approval system might use an ML model to predict credit risk and then apply a rule: "If risk < 0.2 AND income > $50k, then approve."
Traditional XAI methods struggle with these hybrid pipelines. By treating the entire process as a single black box, they often fail to account for the sharp decision boundaries introduced by rules. This paper introduces SMACE (Semi-Model-Agnostic Contextual Explainer), a method specifically engineered to explain these composite structures by leveraging the known geometry of decision rules.
The Architecture of Composite Systems
A composite decision system typically follows a multi-stage architecture. Input features ($x$) are fed into intermediate ML models ($m$), whose outputs are then processed by a decision policy ($P$) to yield a final outcome ($O$).
When rules govern the final decision, small perturbations in input features might result in no change to the output (if the input is far from a threshold) or a massive change (if it is right on the edge). Purely model-agnostic methods often produce "flat" explanations in these scenarios, assigning identical importance to multiple features because they cannot distinguish which feature is the most critical "bottleneck" relative to the rule's thresholds.
The SMACE Methodology
SMACE adopts a "semi-model-agnostic" approach. It treats ML models as black boxes but considers the decision policy (the rules) as a "white box" with a known structure. The algorithm operates in three distinct phases:
- Model Explanation: SMACE utilizes established methods (e.g., KernelSHAP) to determine the local importance of features for each individual ML model in the pipeline.
- Rule Explanation: This is the core innovation. SMACE interprets the decision policy as a set of geometric boundaries. It calculates a contribution score based on the distance between the current input and the decision thresholds.
- Aggregation: The final feature ranking is an aggregation of direct contributions to the rules and indirect contributions through the ML models, weighted by the models' influence on the final decision.
By focusing on the distance to thresholds, SMACE ensures that the magnitude of a feature's importance reflects how sensitive the decision is to that specific variable.
Experimental Results
The researchers evaluated SMACE across several domains, including fraud detection (PaySim), cancer treatment simulation (Breast Cancer Wisconsin), and customer retention (IBM Telco Churn).
Key findings include:
- Ranking Precision: In synthetic tests with simple rules, SMACE correctly ranked features by their proximity to decision boundaries. In contrast, LIME and SHAP often assigned identical scores to different features, failing to provide actionable insights.
- Effective Sanity Checks: SMACE proved superior at identifying "negative features"—the specific inputs that would need to be modified to reverse a decision. In the Cancer Treatment use case, SMACE identified the minimal set of features required to flip a decision more accurately than baseline methods.
Conclusion
SMACE bridges the gap between black-box ML interpretability and transparent business logic. By exploiting the geometric structure of decision rules, it provides more nuanced and intuitive explanations for composite systems. Future research aims to extend SMACE to handle categorical decision rules and recursive systems where models are chained in more complex configurations.
Summary updated