Clinicians increasingly see tabular prediction models (models over rows of labs, vitals, codes and demographics) explained by "shape plots". A shape plot is a curve showing how risk changes as a single variable, such as lactate or age, changes. Recent hybrid architectures make these plots possible without giving up accuracy. They split the model into two parts: an additive part, which is a sum of one-variable curves, and a transformer part, an attention network that captures interactions among variables. The plots describe only the additive part. Nothing guarantees the transformer part behaves the same way for every patient population. It could, for example, learn that "young and female" lowers predicted risk through an interaction that never appears in any curve.
This note proposes to measure that hidden part directly. We would adapt gradient-weighted relevancy propagation, a method that traces how much each input token contributes to a transformer's output by combining attention maps with their gradients, to the feature tokens of a tabular transformer. We would then use it to quantify, separately for each demographic subgroup, how much of the prediction the interaction branch carries and which features drive it. The claim we want to test is that subgroups where the interaction branch does more work are the subgroups where the published shape plots misrepresent the model. We further expect these to be the subgroups where clinically relevant failures concentrate: missed diagnoses and uncertainty intervals that miss their target coverage.
Core Claim
Two hybrid models supply the architecture we audit. Interpretable Additive Tabular Transformer Networks (NATT) sends continuous features through separate one-variable networks and categorical features through a shared transformer. Beyond Black-Box Predictions: Identifying Marginal Feature Effects in Tabular Transformer Networks (NAMformer) runs a shallow additive path alongside a full transformer path. It uses feature dropout so the transformer does not absorb the marginal effects, and it reaches parity with FT-Transformer and XGBoost. Neither paper asks whether the interaction branch encodes population-specific logic. NATT explains categorical importance only through attention to the summary token.
We write the hybrid prediction as $f(x) = a(x) + t(x)$, where $a(x)$ is the additive part shown in shape plots and $t(x)$ is the transformer interaction part. For a subgroup $g$, the interaction share is
$$S_g = \frac{\mathbb{E}_{x \in g}\,|t(x)|}{\mathbb{E}_{x \in g}\,\big(|a(x)| + |t(x)|\big)}.$$
The new contribution is a pair of measurements and a link between them. The first measurement is $S_g$ itself. The second is a gradient-based decomposition of $t(x)$ into feature-token relevancies, computed per subgroup. The link we test is whether between-group differences in these quantities predict two things. The first is explanation infidelity: how far the shape-plot-only risk ranking diverges from the full model's ranking within the group. The second is subgroup harm: the underdiagnosis rate (the false negative rate among truly positive patients) and group-conditional conformal coverage. If the link holds, auditors get a training-free, per-model check for when the interpretable face of a hybrid model cannot be trusted for a given population. If it fails, that is also worth knowing, because it would mean shape plots are a safe clinical explanation even under demographic heterogeneity.
Research Program
The first step is to make gradient relevancy trustworthy on tabular tokens. Transformer Interpretability Beyond Attention Visualization multiplies each layer's attention by its positive gradient, adds the identity to account for skip connections, and chains the layers. It reaches 61.95% mIoU on ImageNet-Segmentation versus 40.82% for Grad-CAM. That evidence comes entirely from vision and text. Tabular tokens are different: each is a projected number or a categorical lookup, there is no spatial structure, and an embedding such as periodic linear encoding can make gradients with respect to the raw value behave unlike gradients with respect to pixels. We would therefore first build synthetic tables with planted structure. Each table has known additive effects, known pairwise interactions, and interactions active only within a planted subgroup. On these tables we check whether relevancy computed on $t(x)$ recovers the planted interacting features better than raw attention to the summary token, vanilla gradients, and Integrated Gradients. This step is a gate. If relevancy does not beat attention on synthetic recovery, the rest of the program falls back to whichever attribution method does.
The second step is the audit itself, run on clinical tables with demographic structure. The main setting is MIMIC-IV in-hospital mortality on the CriticalWindow-24 benchmark used in Predicting ICU In-Hospital Mortality Using Adaptive Transformer Layer Fusion, using only its tabular features. Subgroups are defined by age band, sex and recorded race. The cross-site setting trains on MIMIC and tests on the multi-center eICU-CRD, where institutional coding differences add a second source of population shift. The third setting is the NHANES accelerometry benchmark from Accelerometry-Derived Digital Biomarkers for Cardiometabolic Risk. There, marginal 90% conformal coverage on HbA1c falls to 72% for Mexican American participants, which gives a documented subgroup failure to explain. Age is the primary axis on the strength of Benchmarking Multimodal Clinical Foundation Models to Reveal Significant Demographic Disparities. That paper found age underdiagnosis gaps of 0.32–0.45, far larger than race or gender gaps, though in image-derived embeddings rather than native tabular models. Whether the age effect carries over to tabular hybrids is itself a finding.
For each trained NATT and NAMformer we compute $S_g$, the per-group relevancy profile of $t(x)$ split into demographic tokens and their likely proxies (insurance, admission type, care-unit codes), and explanation infidelity. Infidelity is one minus the within-group rank correlation between $a(x)$ and $f(x)$, together with the within-group AUROC lost by predicting from $a(x)$ alone.
The third step is a targeted intervention that tests causality rather than correlation. If the audit says a group's harm runs through demographic interactions in $t(x)$, then blocking that route should reduce both $S_g$ and the harm. We test two ways of blocking it. One routes demographic features only through the additive path. The other adds a training penalty on the between-group variance of the interaction branch's relevancy mass on demographic and proxy tokens. If harm moves while $S_g$ does not, or the reverse, the audit is not tracking the mechanism.
Baselines
The naive reference point is the shape plots taken at face value. We report subgroup metrics for the full hybrid model and treat the additive curves as its explanation, with no audit. Every claim of explanation infidelity is measured against this reference.
The strongest predictive competitor is FT-Transformer, from Revisiting Deep Learning Models for Tabular Data. It reports an average rank of 1.8 across 11 tabular benchmarks, and on cw-24 it reaches AUPRC 0.566, against 0.577 for an AutoGluon ensemble and 0.585 for ALFIA, all as reported in Predicting ICU In-Hospital Mortality Using Adaptive Transformer Layer Fusion. We found no reported result on Lacuna for NATT or NAMformer on MIMIC-IV. Before auditing, our hybrids must therefore come within about 0.01 AUPRC of 0.566 under the same feature set, split and tuning budget; otherwise we are auditing a model nobody would deploy. On NHANES the reference point is TabPFNv2 at $R^2 = 0.156$ for HbA1c, with its 72% Mexican American coverage.
The strongest competitor for the audit itself is XGBoost with TreeSHAP, including per-group interaction values. It gets the same features and splits, plus a tuning budget equal in wall-clock time to the hybrid's. It beats our proposal if its per-group interaction mass predicts subgroup harm at least as well as $S_g$ plus relevancy does. Two further attribution baselines act on $t(x)$ alone: attention to the summary token as in NATT, and KernelSHAP applied to the interaction branch's output. These isolate the key ingredient, gradient-weighted relevancy, from the branch decomposition itself. For the intervention, the competitor is the Hilbert-Schmidt Independence Criterion (HSIC) penalty, a kernel measure of statistical dependence between learned representations and group labels. FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods found HSIC sits on the best utility-fairness frontier across tabular benchmarks, but that benchmark quotes no scalar we can reuse. We give HSIC the same hyperparameter search as our penalty. Beating it means a lower underdiagnosis gap at matched overall AUPRC, or the same gap with explanations that become more faithful rather than less. The architecture ablation is NAMformer without feature dropout, which tests whether dropout already limits demographic leakage.
Why not just…?
Why not just measure subgroup metrics directly? Subgroup AUROC and coverage show that a gap exists. They cannot show that the explanation shown to clinicians is wrong for that group. Explanation infidelity is a separate quantity, and the program reports it even where performance gaps are small. If $S_g$ adds no predictive value for cross-site harm beyond in-distribution subgroup metrics, the audit offers only diagnosis, not warning, and we would report that.
Why not just use XGBoost with SHAP? This is the most serious threat. Tree models are strong on tables and TreeSHAP interaction values are cheap. The answer is partly about scope: SHAP on a tree model audits a different model, not the hybrid whose shape plots are being shown. It is also partly empirical: we run the head-to-head described above. If tree interaction mass predicts subgroup harm as well as our measure, the honest conclusion is that the phenomenon is about data rather than architecture, and the paper is reframed around that.
Why not just read attention weights, as NATT does? The vision and text evidence in Transformer Interpretability Beyond Attention Visualization is that attention routes information without indicating relevance, and it is the same for every output class. That evidence has not been tested on tabular tokens. The synthetic recovery gate tests it directly, and if attention matches relevancy there, we adopt the cheaper method.
Why not just keep demographics out of the transformer branch? It is one of our two interventions, so the question is whether it suffices. FairPFN: A Tabular Foundation Model for Causal Fairness and Fairer AI in Ophthalmology via Implicit Fairness Learning for Mitigating Sexism and Ageism both report that models recover protected attributes through correlated proxies. Our relevancy profile on proxy tokens shows whether that happens here. If routing alone closes the gaps, that simpler fix is the recommendation.
Proposed Experiments
The first decisive check runs on synthetic tables. Relevancy on $t(x)$ must recover planted subgroup-specific interactions with higher precision than summary-token attention and vanilla gradients. It must also match or beat Integrated Gradients and KernelSHAP at a fraction of their cost. Faithfulness is confirmed with deletion curves that mask the top-ranked tokens and measure the drop in $t(x)$.
The second check is the central hypothesis. Across the 5–7 trained hybrid models per setting, all subgroups, and the three datasets, we test whether $S_g$ and demographic-token relevancy correlate with explanation infidelity and with underdiagnosis-rate gaps. The test is preregistered as a rank correlation with subgroup-bootstrap confidence intervals. The most demanding version is predictive. We compute $S_g$ on MIMIC validation data and ask whether it ranks which groups degrade most on eICU, and, on NHANES, whether it singles out the Mexican American HbA1c coverage failure without being told.
The third check is interventional. Each blocking intervention should lower $S_g$ for the flagged groups and shrink their underdiagnosis or coverage gaps at matched overall AUPRC, compared with HSIC under an equal tuning budget. A negative result at any stage is reported as such, because it bounds when additive shape plots can be trusted.
References
- Interpretable Additive Tabular Transformer Networks
- Beyond Black-Box Predictions: Identifying Marginal Feature Effects in Tabular Transformer Networks
- Transformer Interpretability Beyond Attention Visualization
- Predicting ICU In-Hospital Mortality Using Adaptive Transformer Layer Fusion
- Accelerometry-Derived Digital Biomarkers for Cardiometabolic Risk: A Population-Representative Tabular Benchmark with Uncertainty Quantification
- Benchmarking Multimodal Clinical Foundation Models to Reveal Significant Demographic Disparities
- Revisiting Deep Learning Models for Tabular Data
- FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods
- FairPFN: A Tabular Foundation Model for Causal Fairness
- Fairer AI in ophthalmology via implicit fairness learning for mitigating sexism and ageism