How Much Explanation Leakage Is Just Drift?

Researchers increasingly report that model explanations leak training data. Explanations here are the per-feature importance scores, concept scores, or counterfactuals a system returns to say why it made a decision. The typical evidence is a membership inference attack: a classifier that guesses whether a record was in the training set, fed with explanations instead of model outputs. The same kind of attack on large language models turned out to be largely an illusion. Much of its apparent success came from members and non-members being collected at different times or from different sources, not from anything the model memorized. A classifier that never queries the model can reach an area under the ROC curve (AUC) of 0.99 on WikiMIA. On properly randomized splits, attacks fall to 0.51–0.53 (SoK: Membership Inference Attacks on LLMs are Rushing Nowhere; Blind Baselines Beat Membership Inference Attacks for Foundation Models). Nobody has systematically asked whether explanation-based leakage suffers from the same confound. The question matters most where data drift is unavoidable. Malware detectors are the clearest case: the software they classify changes every month, so training-period samples and later samples always differ. Explanations are also operationally necessary there, because security analysts reject detectors that output bare malicious/benign labels.

We propose a drift-controlled measurement of explanation leakage. An auditor trains a small set of reference models on data from the same period and source as the target's training data, but without the records being audited. The auditor then scores each candidate record by how much more "member-like" the target's explanation looks than the explanations those reference models produce for the same record. Differences in data distribution affect target and reference models alike, so they cancel. Memorization affects only the target, so it survives. This lets us re-measure published explanation-leakage results in settings where a matched non-member set cannot be collected. It also gives the first honest estimate of how much privacy risk explanations add to a drifting security classifier.

Core Claim

Existing explanation-based membership attacks conflate two signals. One is memorization of the specific record. The other is the record's distance from the training distribution in time or source. We claim that a reference-model difference score separates the two. We also predict that in temporally drifting settings, a large share of apparent explanation leakage is drift.

The score is

$$m(x) = s\big(e_f(x)\big) - \frac{1}{K}\sum_{k=1}^{K} s\big(e_{g_k}(x)\big),$$

where $x$ is a candidate record, $f$ is the target model, $g_1,\dots,g_K$ are reference models trained on same-period data excluding $x$, $e_h(x)$ is the explanation model $h$ produces for $x$, and $s$ is any existing attack's membership score computed from an explanation. Two quantities follow. We call the AUC of $s$ applied to reference explanations alone the drift-attributable leakage. We call the AUC of $m$ the memorization-attributable leakage.

Reference-model calibration is a familiar idea for loss-based membership inference. What is new here has three parts. We apply it to explanation outputs. We require the reference models to be time-matched to the target's training window. We use it to decompose already-published explanation-leakage numbers.

Relation to Prior Work

The motivation comes from the language-model auditing literature. Do Membership Inference Attacks Work on Large Language Models? showed that post-cutoff non-members inflate attack scores. The SoK paper and Blind Baselines showed that model-free classifiers beat model-based attacks across nine datasets. Towards More Realistic Membership Inference Attacks on Large Diffusion Models extended the same finding to images. All of these concern loss and confidence signals, not explanations.

On the explanation side, several papers report large leakage under stationary assumptions. Towards a Game-theoretic Understanding of Explanation-based Membership Inference Attacks models explanation variance over repeated queries as a random process and reports 100% attack accuracy on Texas and Purchase. It assumes that variance reflects memorization. High Dimensional Model Explanations: an Axiomatic Approach argues that feature-interaction explanations act as training-data fingerprints. We take these results as claims to be re-tested, not as settled facts.

The closest work is Translating theory into practice. It compared concept-explanation attacks on biomedical models under mismatched and matched non-member sets, and AUC fell from above 80% to below 60%. That paper had the matched set available and treated mismatch as an evaluation flaw to avoid. We target the regime where no matched set exists because the environment itself drifts. We supply an estimator for that regime, and we use their paired result as ground truth to validate it.

Explanation Shift supplies part of our method. It trains a two-sample classifier on SHAP vectors to detect deployment drift. We reuse that detector as a diagnostic, for the reason that makes it dangerous as a privacy measure: it cannot say whether a difference in explanations comes from drift or from memorization. Going Proactive and Explanatory Against Malware Concept Drift establishes that Drebin-to-Malradar drift is severe, which is why we use it as the drift testbed. That paper never asks what its concept explanations leak.

Research Program

The first component is the estimator. We implement $m(x)$ on top of three families of explanation attacks. The first thresholds on attribution variance, following Kumari et al. The second is a learned classifier on raw SHAP or Integrated Gradients vectors. The third scores concept activations, following Lucieri et al. Reference models are trained with the target's recipe on time-matched disjoint data, with $K$ between 4 and 16. We study how the estimate's variance scales with $K$. For hidden commercial models the architecture is unknown, so we also test robustness when the reference architecture differs from the target's. The pairs are LightGBM versus a neural network on EMBER, and ResNet50 versus ConvNeXt on ISIC.

The second component is a re-audit of published claims. For each benchmark where an explanation attack reports high leakage, we report the fraction of above-chance AUC that survives the control.

The third component is the drift regime. We train malware classifiers on early time windows of EMBER and of Drebin followed by Malradar. We audit held-out same-window records against later-window non-members, and measure how drift-attributable leakage grows with the time gap between windows.

Baselines

The naive reference point is the raw explanation attack evaluated on whatever member and non-member sets the setting provides. This reproduces how leakage is currently reported. The strongest published figures are 100% accuracy on Texas/Purchase (Kumari et al.) and above 80% AUC for mismatched concept explanations on ISIC/EyePACS (Lucieri et al.). For explanation-based membership inference on EMBER or Drebin/Malradar, no reported result was found on Lacuna. We establish that number ourselves.

The strongest competitor for separating drift from memorization is the blind baseline. It is a model-free classifier trained on inputs alone: feature-frequency Random Forests for tabular malware features and simple image statistics for ISIC. This follows the bag-of-words and n-gram attacks of Blind Baselines, which reached 94.7% TPR at 5% FPR on WikiMIA against 43.2% for the best model-based attack. The second competitor is sampling tightly around the time cutoff, a regression discontinuity design. The SoK paper showed that this lowers blind AUC from 0.69 to 0.53 on temporal data. Where a matched split exists, as on ISIC/EyePACS, the matched-distribution evaluation of Lucieri et al. serves as an oracle upper bound on correctness. All methods use identical member and non-member sets, the same explanation method, and the same tuning budget. Blind classifiers and attack classifiers each get the same hyperparameter search over a shared held-out fold.

Two ablations isolate the key ingredient. The first uses reference models trained on randomly mixed-period data instead of time-matched data. If the estimator is working, performance should degrade toward the naive attack under drift. The second replaces reference explanations with a blind input-only score subtracted in the same way, which tests whether the explanation channel matters at all.

Beating the baselines means the following. On settings with an oracle, the controlled AUC should land closer to the oracle than both the blind-adjusted score and the time-window-matched score. On drift settings, the control should agree with time-window matching where that is feasible, while retaining the full sample instead of a narrow window.

Why not just…?

Why not just run a blind baseline and stop? A blind classifier proves a shift exists, but it cannot say how much of the attack's success is due to that shift. When blind AUC is 0.75 and the explanation attack's AUC is 0.85, the memorization share is undetermined, because the two signals can overlap or add. Blind baselines also collapse to near chance on randomized splits (0.51–0.53 on The Pile, per the SoK paper). They are silent exactly where residual memorization might exist. Our second ablation directly compares blind subtraction with reference-explanation subtraction.

Why not just sample non-members tightly around the cutoff? This is a real threat, and where it is feasible it may match our estimator. However, malware drift is continuous rather than a single change point. Narrow windows shrink the audit set. The SoK paper also found that cleaning steps such as aggressive deduplication can create new shifts. We run cutoff matching at several window widths on EMBER and Drebin/Malradar. If it matches the reference control at usable sample sizes, we will report cutoff matching as sufficient and restrict our method to settings without time stamps, such as source shifts like ACS state splits.

Why not just train the target with differential privacy? Differential privacy bounds leakage but does not measure it, and an auditor of a deployed model cannot retrain that model. Lucieri et al. also found that differential privacy harmed accuracy and explanation fidelity, and increased vulnerability on imbalanced data. For defending explanations specifically, XRand is the more relevant tool. Our measurement is what tells a deployer whether such a defense is needed at all.

Why not just use the Explanation Shift detector to compare member and non-member explanations? Its two-sample classifier fires on any difference in explanations, whether from drift or from memorization. Using it as a privacy metric would reproduce the confound in explanation space. We include it as a diagnostic and expect it to track drift-attributable rather than memorization-attributable leakage.

Proposed Experiments

The decisive validation uses ISIC and EyePACS, where both settings exist. Applied to the mismatched setting, our controlled score should recover an AUC near the matched-setting value of below 60% reported by Lucieri et al., while the raw attack stays above 80%. If it does not, the estimator fails.

On Texas and Purchase we build source-shifted non-member sets by holding out subgroups. We then test whether the 100% attack accuracy survives the control. Our prediction is that it survives under the original random splits but mostly vanishes under shifted splits.

On EMBER and Drebin/Malradar, with LightGBM and neural classifiers explained by SHAP and Integrated Gradients, we report raw, blind, cutoff-matched, and controlled AUC plus TPR at 1% FPR. Each is plotted against the time gap between member and non-member windows. The central prediction is that raw explanation leakage grows with the gap while controlled leakage stays flat.

A final check covers failure modes. We deliberately insert repeated canary records into training, following the copyright-trap design discussed in the SoK paper. The control must still detect these canaries; if it does not, it is over-subtracting and removing real memorization.