TL;DR
Post-hoc explainability methods like SHAP can be computationally sluggish and susceptible to misleading feature-perturbation artifacts, while existing self-interpreting neural networks often sacrifice predictive accuracy. SASANet resolves this trade-off by intrinsically computing Shapley values via an internal distillation schema over variable-sized feature subsets, matching black-box model performance while delivering instant, theoretically guaranteed Shapley attributions in a single forward pass.
What Problem Does This Paper Address?
Explaining complex neural networks usually follows one of two routes:
- Post-hoc attribution methods (e.g., KernelSHAP, LIME): These approximate feature importance by masking or replacing features. However, simulating missing inputs using marginal sampling often fabricates out-of-distribution feature combinations, distorting attributions when features are correlated. They are also computationally expensive at inference time.
- Inherently interpretable models (e.g., SENN, Neural Additive Models [NAM]): These enforce additive or prototype structures to provide explanations by design, but restrictive structural assumptions or regularizations often degrade predictive performance.
The authors tackle the challenge of designing an intrinsically interpretable neural network that satisfies the axiomatic fairness of Shapley values from cooperative game theory without sacrificing expressive capacity or speed.
Methods
The authors introduce the Additive Self-Attribution (ASA) framework and implement it via SASANet:
- Set-Based Value Function ($f_c$): Unlike standard networks that require fixed-dimension inputs, SASANet directly accepts arbitrary subsets of features $x_S$. An autoregressive marginal contribution module uses multi-head attention with position embeddings to compute each feature's contribution given an ordering: $$\Delta(x_{O_i}, x_{O_{1:i-1}}; \theta_\Delta)$$
- Subset Expectation Supervision ($L_v$): Intermediate outputs are trained across random feature permutations to approximate the true conditional label expectation $\mathbb{E}[y | x_S]$, grounding the value function in actual feature-label relationships rather than arbitrary synthetic reference values.
- Positional Shapley Value Distillation ($L_s$): Rather than directly enforcing permutation invariance, a dedicated permutation-invariant Shapley module learns feature attributions $\phi(x_S)_i$. To stabilize convergence and minimize variance, it distills position-specific marginal contributions $\phi(x_S)_{i,k}$ before averaging across positions $k$.
- Theoretical Guarantees: The authors mathematically prove that optimizing this distillation objective guarantees that the intrinsic self-attribution values converge to the exact Shapley values of the model’s predictions, satisfying the Efficiency, Linearity, Nullity, and Symmetry axioms.
[ Input Features x ]
│
┌──────┴─────────────────────────────────┐
│ (Sequential Module + Permutation O) │ (Shapley Value Module)
▼ ▼
Marginal Contributions Δ(x_{O_k}, ...) Positional Attributions ϕ_i,k
│ │
│ ─── Internal Distillation (Ls) ──────► │
▼ ▼
Accumulated Output f_c(x, O) Permutation-Invariant f(x) = ∑ ϕ_i
Datasets
The authors evaluate SASANet on standardized tabular benchmarks:
- Census Income Prediction: Binary classification predicting income above \$50K (39 features, ~299K samples).
- Higgs Boson Identification: High-energy physics binary classification (28 kinematic features, 10.5M samples).
- Credit Card Fraud Detection: Highly imbalanced transaction classification (30 PCA-transformed features, ~285K samples).
- Insurance Company Benchmark (CoIL 2000): Continuous regression task predicting customer product usage (85 features, 5.8K samples).
No new benchmarks were introduced.
Key Insights and Ideas
- Bridging the Accuracy–Interpretability Gap: SASANet functions as an implicit ensemble (bagging) over all permutation orders of sequential marginal contributions. Consequently, it achieves performance rivaling unconstrained MLPs and LightGBM models.
- Natural Handling of Feature Dependencies: By training the underlying sequential module on variable-sized feature subsets with real label expectations, SASANet avoids the out-of-distribution data synthesis that causes post-hoc methods like KernelSHAP to assign exaggerated, mutually canceling weights to correlated features.
- Positional Stratification Reduces Variance: Learning marginal contributions conditioned on their position index $k$ in the sequence mathematically lowers the estimation variance compared to directly regressing an overall Shapley value.
Key Results
- Predictive Performance: SASANet achieved Average Precision (AP) and AUC scores superior to all existing interpretable baselines (NAM, SITE, SENN) and competitive with black-box models:
- Income: SASANet AP 0.6864 vs. LightGBM 0.6972, NAM 0.6567, SENN 0.6067.
- Higgs Boson: SASANet AP 0.8836 vs. MLP 0.8877, LightGBM 0.8590, NAM 0.7897.
- Fraud: SASANet AP 0.8124 vs. MLP 0.8167, LightGBM 0.7934.
- Inference Speed: Explaining 1,000 samples took 0.2 to 0.3 seconds for SASANet, compared to 38.4 to 91.2 seconds for KernelSHAP (>200× slower) and 1,600 to 11,800 seconds for LIME.
- Faithfulness (Feature Masking): When progressively masking the top 1 to 5 attributed features, SASANet degraded performance significantly faster than KernelSHAP or FastSHAP (e.g., Higgs Boson AUC dropped to 0.616 with SASANet vs. 0.746 with KernelSHAP), proving its attributions more faithfully track the true decision logic.
- Shapley Approximation Error: Compared to ground-truth Shapley values computed over 10,000 permutations, SASANet attained an RMSE of 0.001 to 0.033, outperforming KernelSHAP (0.348 to 0.514).
Limitations and Future Work
- Distribution Shifts: Under artificial distribution shifts (e.g., large input biases), SASANet’s attribution RMSE rose (up to 0.113 on Higgs), reflecting that distillation is tied to the training distribution. The authors suggest injecting synthetic noise during internal distillation as a countermeasure.
- Tabular Data Focus: The empirical validation focuses entirely on tabular data. While the paper outlines how SASANet could incorporate image feature extractors for concept-level attribution, scaling to high-dimensional unstructured modalities (pixels, raw text tokens) remains untested.
- Permutation Sampling at Training: While single-permutation sampling per batch is efficient, training still requires learning representations across combinatorial permutations of subsets, which may increase training time on datasets with very large feature counts.
Summary updated